Oct 30 12:29:31.681756 ip-10-0-136-246 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 12:29:31.681769 ip-10-0-136-246 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 12:29:31.681776 ip-10-0-136-246 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 12:29:31.682076 ip-10-0-136-246 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 12:29:42.918350 ip-10-0-136-246 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 12:29:42.918370 ip-10-0-136-246 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 72a3a50f696b4435a83837eb3eb12fa5 --
Oct 30 12:31:41.415600 ip-10-0-136-246 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:31:41.881147 ip-10-0-136-246 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:31:41.881147 ip-10-0-136-246 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:31:41.881147 ip-10-0-136-246 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:31:41.881147 ip-10-0-136-246 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:31:41.886102 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886000    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:41.886102 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886098    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:41.886102 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886102    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:41.886102 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886106    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:41.886102 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886109    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:41.886102 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886112    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886115    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886118    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886121    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886124    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886126    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886129    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886132    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886134    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886137    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886140    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886143    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886145    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886147    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886150    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886152    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886155    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886158    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886177    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886180    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:41.886365 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886182    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886185    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886188    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886191    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886193    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886196    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886199    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886201    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886204    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886206    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886208    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886211    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886214    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886216    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886220    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886222    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886225    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886228    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886230    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:41.886833 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886233    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886237    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886243    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886246    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886249    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886252    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886254    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886258    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886261    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886264    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886266    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886268    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886271    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886274    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886276    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886279    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886295    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886298    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886300    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886303    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:41.887337 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886306    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886308    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886311    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886313    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886316    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886319    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886321    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886325    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886329    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886334    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886337    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886340    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886343    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886345    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886348    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886351    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886353    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886356    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886358    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:41.887814 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886362    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886365    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886368    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886371    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886375    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886377    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886380    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886383    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886386    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886388    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886391    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886393    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886396    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886399    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886401    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886404    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886406    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886409    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886411    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886414    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:41.888265 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886418    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886421    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.886423    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887223    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887232    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887235    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887238    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887240    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887243    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887246    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887248    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887251    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887253    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887257    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887260    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887262    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887265    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887267    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887270    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:41.888761 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887273    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887275    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887278    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887297    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887300    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887303    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887305    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887308    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887311    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887313    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887316    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887318    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887321    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887324    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887327    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887330    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887332    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887335    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887337    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:41.889225 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887340    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887343    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887346    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887348    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887351    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887353    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887356    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887359    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887362    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887365    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887367    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887370    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887372    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887374    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887377    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887380    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887383    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887385    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887388    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887391    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:41.889732 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887393    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887396    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887398    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887401    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887403    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887406    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887408    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887411    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887413    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887415    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887418    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887421    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887423    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887425    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887428    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887431    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887433    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887435    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887438    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:41.890195 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887440    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887446    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887450    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887453    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887456    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887458    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887461    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887463    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887466    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887468    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887476    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887478    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887481    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887483    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887486    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887488    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887491    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887493    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887496    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887499    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:41.890664 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887501    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887504    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887507    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887509    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887512    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887514    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887517    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887519    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887522    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887524    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887527    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.887529    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888105    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888113    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888117    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888121    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888123    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888126    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888128    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:41.891142 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888131    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888134    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888138    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888143    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888146    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888148    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888151    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888153    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888156    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888159    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888161    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888164    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888167    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888169    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888172    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888174    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888177    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888180    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888182    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888185    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:41.891600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888188    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888190    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888193    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888195    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888198    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888200    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888203    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888206    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888209    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888211    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888214    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888217    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888219    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888222    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888225    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888228    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888231    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888233    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888236    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888239    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:41.892077 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888242    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888245    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888248    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888250    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888253    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888255    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888258    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888261    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888265    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888269    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888272    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888276    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888279    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888299    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888302    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888305    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888307    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888310    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888313    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:41.892563 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888315    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888318    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888322    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888324    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888327    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888330    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888332    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888335    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888339    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888342    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888344    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888347    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888349    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888352    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888354    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888357    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888360    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888362    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888364    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:41.893000 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888367    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888369    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888372    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888374    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888377    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888380    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888382    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888384    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888387    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888389    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888393    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888395    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888398    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888400    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888403    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888405    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888408    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888411    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888413    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:41.893475 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888416    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.888418    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889458    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889471    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889479    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889484    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889489    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889492    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889497    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889501    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889505    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889508    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889511    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889515    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889518    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889521    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889524    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889527    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889530    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889533    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889537    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889540    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889543    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889546    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:31:41.893929 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889549    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889554    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889557    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889560    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889564    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889567    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889570    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889574    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889577    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889580    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889586    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889589    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889592    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889595    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889598    2571 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889601    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889606    2571 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889609    2571 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889612    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889615    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889618    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889622    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889625    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889631    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889635    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:31:41.894510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889638    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889641    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889645    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889648    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889651    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889654    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889657    2571 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889661    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889664    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889668    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889671    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889674    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889677    2571 flags.go:64] FLAG: --help="false"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889680    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-136-246.ec2.internal"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889684    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889687    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889690    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889694    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889697    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889700    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889703    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889706    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889709    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:31:41.895122 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889711    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889715    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889717    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889721    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889724    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889727    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889729    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889733    2571 flags.go:64] FLAG: --lock-file=""
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889736    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889740    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889743    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889748    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889752    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889755    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889758    2571 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889761    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889764    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889767    2571 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889770    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889775    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889778    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889782    2571 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889785    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889788    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889791    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:31:41.895873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889794    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889798    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889801    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889804    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889812    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889816    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889819    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889822    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889825    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889828    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889831    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889834    2571 flags.go:64] FLAG: --port="10250"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889837    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889840    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0423db029ce944085"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889844    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889847    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889850    2571 flags.go:64] FLAG: --register-node="true"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889854    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889857    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889860    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889863    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889866    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889870    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889874    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889877    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889880    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:31:41.896493 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889883    2571 flags.go:64] FLAG: --runonce="false"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889886    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889889    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889892    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889895    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889898    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889902    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889906    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889909    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889912    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889915    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889918    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889921    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889924    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889927    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889930    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889935    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889938    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889941    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889944    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889949    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889952    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889955    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889958    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889961    2571 flags.go:64] FLAG: --v="2"
Oct 30 12:31:41.897392 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889966    2571 flags.go:64] FLAG: --version="false"
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889970    2571 flags.go:64] FLAG: --vmodule=""
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889974    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.889977    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890115    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890121    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890124    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890127    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890130    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890133    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890135    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890138    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890141    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890143    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890146    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890149    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890152    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890154    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890157    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890159    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:41.898465 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890162    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890164    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890167    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890170    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890172    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890175    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890177    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890182    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890184    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890186    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890189    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890192    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890194    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890197    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890199    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890202    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890206    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890211    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890215    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:41.899343 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890218    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890220    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890223    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890226    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890228    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890231    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890233    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890236    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890242    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890245    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890248    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890251    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890253    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890256    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890258    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890261    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890264    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890268    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890271    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:41.900204 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890274    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890278    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890294    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890297    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890300    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890303    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890306    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890308    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890311    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890313    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890316    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890320    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890323    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890325    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890328    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890330    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890333    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890336    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890338    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:41.901056 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890340    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890343    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890346    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890349    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890352    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890355    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890357    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890360    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890363    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890365    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890368    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890371    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890373    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890376    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890378    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890381    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890383    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890386    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890388    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890391    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:41.901927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890393    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890396    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890398    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890401    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890405    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890408    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890410    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890413    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890416    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890418    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890421    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890423    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890426    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.890454    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890511    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890516    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:41.902561 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890520    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890524    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890527    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890530    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890533    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890535    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890538    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890540    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890543    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890545    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890548    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890550    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890552    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890555    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890558    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890560    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890563    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890566    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890568    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890571    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:41.902969 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890574    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890578    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890581    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890583    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890586    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890589    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890591    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890594    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890596    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890599    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890603    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890621    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890624    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890629    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890633    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890636    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890638    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890641    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890644    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890646    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:41.903805 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890649    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890651    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890653    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890656    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890659    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890661    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890664    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890677    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890681    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890684    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890688    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890692    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890694    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890698    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890702    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890704    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890708    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890711    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890713    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:41.904600 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890716    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890718    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890721    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890723    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890726    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890728    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890731    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890734    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890737    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890739    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890742    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890744    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890747    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890749    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890752    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890754    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890757    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890760    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890762    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:41.905043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890765    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890767    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890770    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890772    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890775    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890777    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890780    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890784    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890786    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890789    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890792    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890795    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890798    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890800    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890803    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890805    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890808    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890811    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890814    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890816    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:41.905776 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890819    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890823    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890826    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890829    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890831    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.890834    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.899825    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.899847    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.899872    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.899879    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.899988    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.899996    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900001    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900005    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900010    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900014    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900018    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900023    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900027    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:41.906249 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900031    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900035    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900039    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900044    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900049    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900053    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900058    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900062    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900067    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900071    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900075    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900080    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900085    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900089    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900095    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900099    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900104    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900108    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900112    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900117    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:41.906767 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900122    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900126    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900130    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900137    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900143    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900148    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900153    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900157    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900161    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900165    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900170    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900174    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900179    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900183    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900187    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900191    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900195    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900200    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900204    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900208    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:41.907245 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900212    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900216    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900220    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900225    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900229    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900233    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900238    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900243    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900247    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900252    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900256    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900261    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900265    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900270    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900274    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900278    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900301    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900305    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900310    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900315    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:41.907836 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900319    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900326    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900333    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900338    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900343    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900347    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900352    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900356    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900360    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900364    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900369    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900373    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900377    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900381    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900385    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900390    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900394    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900398    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900402    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:41.908396 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900407    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900412    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900416    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900420    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900425    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900430    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900434    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900438    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900442    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900446    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900451    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900455    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900459    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900463    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900467    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900471    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900475    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:41.909043 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900479    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.900522    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900841    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900851    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900857    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900861    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900866    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900871    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900875    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900879    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900884    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900888    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900892    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900897    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900901    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900905    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:41.909741 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900909    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900915    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900919    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900924    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900928    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900932    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900937    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900942    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900947    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900951    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900955    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900960    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900964    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900968    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900972    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900976    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900980    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900985    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900989    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:41.910426 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900993    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.900998    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901002    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901006    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901010    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901015    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901019    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901023    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901029    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901033    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901037    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901042    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901046    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901050    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901057    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901066    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901086    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901093    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901098    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:41.911082 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901102    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901107    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901111    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901115    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901120    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901124    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901128    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901132    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901137    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901141    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901145    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901149    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901154    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901158    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901162    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901167    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901171    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901175    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901179    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901183    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:41.911736 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901187    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901193    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901197    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901201    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901205    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901209    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901213    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901217    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901222    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901231    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901236    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901241    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901246    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901250    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901254    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901258    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901262    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901267    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901271    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:41.912479 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901275    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901295    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901301    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901305    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901309    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901313    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901317    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901322    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901326    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901330    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901334    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901339    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901343    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901346    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901351    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.901397    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:31:41.913074 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901472    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901480    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901486    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901491    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901495    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901499    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901503    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901513    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901518    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901523    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901527    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901532    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901536    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901540    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901544    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901548    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901553    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901557    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901561    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:31:41.913500 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901566    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901570    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901574    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901579    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901583    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901588    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901591    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901596    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901600    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901604    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901608    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901612    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901617    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901621    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901628    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901635    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901641    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901647    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901652    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:31:41.913944 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901656    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901661    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901673    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901677    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901682    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901686    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901690    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901694    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901699    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901703    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901707    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901712    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901716    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901720    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901724    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901728    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901733    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901737    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901742    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901746    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:31:41.914409 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901750    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901756    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901760    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901767    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901771    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901776    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901781    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901785    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901789    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901794    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901798    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901802    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901806    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901810    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901814    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901829    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901834    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901838    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901842    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:31:41.914871 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901846    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901850    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901854    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901860    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901864    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901868    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901872    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901876    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901880    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901884    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901889    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901893    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901897    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901902    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901906    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901910    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901916    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901920    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901924    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901935    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:31:41.915332 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901939    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901943    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901947    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901951    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901955    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901959    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901963    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901967    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:41.901972    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.902073    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.905321    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.906545    2571 server.go:1014] "Starting client certificate rotation"
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.906649    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.906695    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.909240    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:31:41.915834 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.913377    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:31:41.927347 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.927329    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:31:41.931452 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.931434    2571 log.go:25] "Validated CRI v1 image API"
Oct 30 12:31:41.932371 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.932349    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:31:41.937037 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.937004    2571 fs.go:135] Filesystem UUIDs: map[0d3af223-41f0-4441-8f69-dd57254ecba2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fc874164-8534-4914-979b-485db38c5142:/dev/nvme0n1p3]
Oct 30 12:31:41.937125 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.937035    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 12:31:41.941452 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.941338    2571 manager.go:217] Machine: {Timestamp:2026-10-30 12:31:41.940827368 +0000 UTC m=+0.392590497 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3073872 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec248f378efa70f7ff5f699330a040fd SystemUUID:ec248f37-8efa-70f7-ff5f-699330a040fd BootID:72a3a50f-696b-4435-a838-37eb3eb12fa5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:0f:c8:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:0f:c8:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:6c:0e:f3:86:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:31:41.941452 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.941452    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:31:41.941589 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.941576    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:31:41.942796 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.942770    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:31:41.942940 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.942798    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-246.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:31:41.942985 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.942948    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:31:41.942985 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.942957    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:31:41.942985 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.942971    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:31:41.944032 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.944021    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:31:41.945386 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.945375    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:31:41.950890 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.950877    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:31:41.951008 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.950999    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:31:41.951966 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.951956    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:31:41.952007 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.951973    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:31:41.952007 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.951986    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:31:41.952007 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.951995    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:31:41.952007 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.952005    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:31:41.952457 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.952440    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:31:41.955050 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.955026    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:31:41.962440 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.962422    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:31:41.963407 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.963393    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:31:41.963448 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.963440    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:31:41.963652 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.963636    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:31:41.963689 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.963666    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:31:41.963689 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.963681    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:31:41.963740 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.963695    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:31:41.963740 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.963706    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:31:41.963740 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.963715    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:31:41.964401 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.964393    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:31:41.964430 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.964402    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:31:41.964430 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.964409    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:31:41.964430 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.964415    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:31:41.964430 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.964424    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:31:41.964550 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.964434    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:31:41.965896 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.965885    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:31:41.965930 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.965900    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:31:41.970049 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.970035    2571 server.go:1263] "Started kubelet"
Oct 30 12:31:41.974195 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.974088    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:31:41.974744 ip-10-0-136-246 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:31:41.975531 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.975337    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:31:41.975531 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.975474    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:31:41.977520 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.977497    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:31:41.977851 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.977813    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:31:41.982303 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:41.982257    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 12:31:41.985089 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.985059    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:31:41.985744 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.985667    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:31:41.985912 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:41.985845    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:41.986419 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.986402    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:31:41.986419 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.986421    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:31:41.986602 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.986590    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:31:41.986704 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.986691    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:31:41.986768 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.986717    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:31:41.987265 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.987245    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:31:41.987265 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.987267    2571 factory.go:55] Registering systemd factory
Oct 30 12:31:41.987394 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.987277    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:31:41.987574 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.987549    2571 factory.go:153] Registering CRI-O factory
Oct 30 12:31:41.987640 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.987579    2571 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:31:41.987640 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.987599    2571 factory.go:103] Registering Raw factory
Oct 30 12:31:41.987640 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.987614    2571 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:31:41.988875 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.988860    2571 manager.go:319] Starting recovery of all containers
Oct 30 12:31:41.998585 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:41.998571    2571 manager.go:324] Recovery completed
Oct 30 12:31:42.002454 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.002413    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:42.003875 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.003854    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-246.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 12:31:42.003983 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.003856    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 12:31:42.004486 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.004468    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:42.004559 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.004508    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:42.004559 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.004524    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:42.005106 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.003852    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-246.ec2.internal.18e34e9a781296c5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-246.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-246.ec2.internal,},FirstTimestamp:2026-10-30 12:31:41.970003653 +0000 UTC m=+0.421766782,LastTimestamp:2026-10-30 12:31:41.970003653 +0000 UTC m=+0.421766782,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-246.ec2.internal,}"
Oct 30 12:31:42.005179 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.005157    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:31:42.005179 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.005164    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:31:42.005246 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.005180    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:31:42.005246 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.005223    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:42.005994 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.005980    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:42.006047 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.006009    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:42.006454 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.006429    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:42.008303 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.008272    2571 policy_none.go:50] "Start"
Oct 30 12:31:42.008545 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.008525    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:31:42.008945 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.008934    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:31:42.008986 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.008953    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:31:42.010986 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.010973    2571 policy_none.go:44] "Start"
Oct 30 12:31:42.013347 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.013332    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vhv26"
Oct 30 12:31:42.019445 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.019315    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 12:31:42.019525 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.019315    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 12:31:42.023417 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.023399    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vhv26"
Oct 30 12:31:42.035902 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.035878    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 12:31:42.035972 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.035831    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-246.ec2.internal.18e34e9a7a20c84d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-246.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-246.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-246.ec2.internal,},FirstTimestamp:2026-10-30 12:31:42.004488269 +0000 UTC m=+0.456251403,LastTimestamp:2026-10-30 12:31:42.004488269 +0000 UTC m=+0.456251403,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-246.ec2.internal,}"
Oct 30 12:31:42.047847 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.047814    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:31:42.050578 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.050565    2571 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:31:42.050629 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.050598    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:31:42.050629 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.050607    2571 server.go:87] "Starting device plugin registration server"
Oct 30 12:31:42.050828 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.050819    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:31:42.050863 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.050830    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:31:42.050959 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.050935    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:31:42.051062 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.051027    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:31:42.051062 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.051037    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:31:42.051530 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.051508    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:31:42.051609 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.051558    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.119133 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.119105    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:31:42.121156 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.120445    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:31:42.121156 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.120465    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:31:42.121156 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.120488    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:31:42.121156 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.120523    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 12:31:42.133175 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.133123    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:42.151863 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.151846    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:42.152828 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.152811    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:42.152904 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.152848    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:42.152904 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.152865    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:42.152904 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.152892    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.174618 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.174595    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.174705 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.174622    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-246.ec2.internal\": node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.206392 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.206363    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.221452 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.221421    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"]
Oct 30 12:31:42.221550 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.221503    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:42.222546 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.222522    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:42.222657 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.222557    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:42.222657 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.222570    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:42.224873 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.224856    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:42.225035 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.225019    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.225094 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.225054    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:42.225686 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.225669    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:42.225686 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.225682    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:42.225810 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.225698    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:42.225810 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.225711    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:42.225896 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.225702    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:42.225896 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.225852    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:42.227862 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.227849    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.227944 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.227873    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:42.228539 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.228524    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:42.228633 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.228549    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:42.228633 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.228565    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:42.268013 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.267990    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-246.ec2.internal\" not found" node="ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.270072 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.270052    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-246.ec2.internal\" not found" node="ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.306610 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.306581    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.387582 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.387490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9b5f11662f569a9096e79917d2f933f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-246.ec2.internal\" (UID: \"e9b5f11662f569a9096e79917d2f933f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.387582 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.387533    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.387582 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.387550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.407673 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.407642    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.488359 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.488331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9b5f11662f569a9096e79917d2f933f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-246.ec2.internal\" (UID: \"e9b5f11662f569a9096e79917d2f933f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.488480 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.488364    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.488480 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.488382    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.488480 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.488451    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.488480 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.488464    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9b5f11662f569a9096e79917d2f933f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-246.ec2.internal\" (UID: \"e9b5f11662f569a9096e79917d2f933f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.488636 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.488520    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.508068 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.508037    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.572504 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.572466    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.573530 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.573509    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:42.608369 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.608327    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.708901 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.708805    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.809301 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.809251    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:42.871151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.871123    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:42.906504 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.906469    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 12:31:42.906954 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.906638    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:31:42.906954 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:42.906647    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:31:42.909601 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:42.909575    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:43.010197 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:43.010167    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:43.029351 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.029310    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:26:42 +0000 UTC" deadline="2028-04-01 20:54:49.752528068 +0000 UTC"
Oct 30 12:31:43.029351 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.029345    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12464h23m6.723186525s"
Oct 30 12:31:43.041818 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.041791    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:31:43.043064 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.043037    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:31:43.043168 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.043089    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:31:43.043168 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.043106    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:31:43.110989 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:43.110961    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:43.115102 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:43.115062    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b5f11662f569a9096e79917d2f933f.slice/crio-55f1176799472fe09027b81fa69da97e56aeafd1bfeb5b08ec65d48961c1f2bd WatchSource:0}: Error finding container 55f1176799472fe09027b81fa69da97e56aeafd1bfeb5b08ec65d48961c1f2bd: Status 404 returned error can't find the container with id 55f1176799472fe09027b81fa69da97e56aeafd1bfeb5b08ec65d48961c1f2bd
Oct 30 12:31:43.115541 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:43.115520    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7fe78ff9a981c06dfba04265b1aa94.slice/crio-eb77d6c826e15d3b10bafd03931dde3e18da3def548763d58e124f21237f010d WatchSource:0}: Error finding container eb77d6c826e15d3b10bafd03931dde3e18da3def548763d58e124f21237f010d: Status 404 returned error can't find the container with id eb77d6c826e15d3b10bafd03931dde3e18da3def548763d58e124f21237f010d
Oct 30 12:31:43.119634 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.119618    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:31:43.125388 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.124014    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal" event={"ID":"e9b5f11662f569a9096e79917d2f933f","Type":"ContainerStarted","Data":"55f1176799472fe09027b81fa69da97e56aeafd1bfeb5b08ec65d48961c1f2bd"}
Oct 30 12:31:43.126623 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.126593    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal" event={"ID":"3a7fe78ff9a981c06dfba04265b1aa94","Type":"ContainerStarted","Data":"eb77d6c826e15d3b10bafd03931dde3e18da3def548763d58e124f21237f010d"}
Oct 30 12:31:43.196778 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.196738    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:43.211807 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:43.211774    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:43.312377 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:43.312299    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:43.412980 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:43.412938    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:43.513710 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:43.513675    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:43.614420 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:43.614340    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:31:43.634672 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.634642    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:43.687716 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.687676    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:43.716296 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.716252    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:31:43.717132 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.717101    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:31:43.722492 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.722467    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:43.738718 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.738690    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:31:43.953705 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.953615    2571 apiserver.go:52] "Watching apiserver"
Oct 30 12:31:43.987050 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.987016    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 12:31:43.997103 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.997074    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:31:43.997568 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:43.997539    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-2s9ms","kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lgqrc","openshift-multus/multus-k4c5j","openshift-multus/network-metrics-daemon-kfgnl","kube-system/global-pull-secret-syncer-6h57s","kube-system/konnectivity-agent-xz8gl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn","openshift-image-registry/node-ca-r8lf8","openshift-network-diagnostics/network-check-target-cqc2z","openshift-dns/node-resolver-fttst","openshift-multus/multus-additional-cni-plugins-fpqzd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal","openshift-network-operator/iptables-alerter-qmdgt"]
Oct 30 12:31:44.000383 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.000337    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:44.002830 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.002806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.005700 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.005257    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.007678 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.007624    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.010454 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.010433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:44.010556 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.010521    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:44.013015 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.012707    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.013015 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.012819    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:44.015842 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.014988    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.018633 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.018616    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56d7b\""
Oct 30 12:31:44.018794 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.018728    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:31:44.018794 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.018748    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:31:44.018921 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.018843    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.019791 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.019773    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v9g2h\""
Oct 30 12:31:44.019877 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.019851    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:31:44.019877 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.019852    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:31:44.020047 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.020025    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.021719 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.021377    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k87jk\""
Oct 30 12:31:44.021719 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.021472    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:31:44.021719 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.021537    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:31:44.023029 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.022655    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:44.023029 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.022745    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:44.023452 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.023432    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:31:44.024814 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.024791    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:31:44.024967 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.024947    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:31:44.025043 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.025008    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:31:44.025107 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.025071    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fkzfg\""
Oct 30 12:31:44.026603 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.025912    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.028796 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.028725    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.087995 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.087970    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:31:44.097593 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-modprobe-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.097736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-lib-modules\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.097736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097639    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-tmp\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.097736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097709    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-registration-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.097923 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-device-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.097923 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097796    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-cnibin\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.097923 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-ovn\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.097923 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097894    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-bin\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.097923 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097919    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-kubelet\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.098084 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.097986    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-daemon-config\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.098084 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098034    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:44.098151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098068    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.098151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098116    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-env-overrides\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.098151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.098251 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-sys-fs\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.098251 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7dm\" (UniqueName: \"kubernetes.io/projected/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kube-api-access-7w7dm\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.098426 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:44.098426 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-slash\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.098426 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-var-lib-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.098426 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-log-socket\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.098653 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098447    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-system-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.098653 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098492    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-socket-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.098653 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-systemd\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.098653 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098544    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-script-lib\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.098653 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztsmb\" (UniqueName: \"kubernetes.io/projected/4825a637-bc1a-4b66-acd1-c0081284ae23-kube-api-access-ztsmb\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.098653 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-etc-kubernetes\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.098892 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098692    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc2566d-5249-44ca-be3a-697d3eb4d603-host-slash\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.098892 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-etc-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.098892 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098770    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-system-cni-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.098892 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-netns\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.098892 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-conf-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.099151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqfh\" (UniqueName: \"kubernetes.io/projected/adf5def0-890c-405c-9706-664675f53547-kube-api-access-9dqfh\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.099151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098936    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.099151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.098981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cee19c40-c94c-45cd-8deb-16e8a965e053-konnectivity-ca\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:44.099151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099012    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4825a637-bc1a-4b66-acd1-c0081284ae23-hosts-file\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.099151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxsh\" (UniqueName: \"kubernetes.io/projected/f2102d05-9eb2-4641-a057-7b02ae7d88bd-kube-api-access-rtxsh\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:44.099151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099080    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-kubernetes\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.099151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-socket-dir-parent\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.099151 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099146    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-kubelet-config\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099184    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-node-log\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099238    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-systemd-units\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099300    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-multus\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099334    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-host\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099360    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-etc-tuned\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-systemd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099412    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-config\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.099577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099556    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cnibin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099583    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjh8\" (UniqueName: \"kubernetes.io/projected/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-kube-api-access-cmjh8\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-dbus\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099644    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf513a7-048e-42d8-98fb-5b7adbc09011-host\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099674    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099699    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cni-binary-copy\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099781    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-k8s-cni-cncf-io\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099814    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysconfig\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-run\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099876    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-netns\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099903    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099958    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.099993    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-multus-certs\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.100024 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75mt\" (UniqueName: \"kubernetes.io/projected/4bc2566d-5249-44ca-be3a-697d3eb4d603-kube-api-access-j75mt\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-var-lib-kubelet\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-kubelet\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100130    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-netd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100157    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovn-node-metrics-cert\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100183    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7cq\" (UniqueName: \"kubernetes.io/projected/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-kube-api-access-6w7cq\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100288    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-bin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100319    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-conf\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100343    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-sys\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8h5\" (UniqueName: \"kubernetes.io/projected/6bf513a7-048e-42d8-98fb-5b7adbc09011-kube-api-access-zx8h5\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100415    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100447    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100488    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cee19c40-c94c-45cd-8deb-16e8a965e053-agent-certs\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100540    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bc2566d-5249-44ca-be3a-697d3eb4d603-iptables-alerter-script\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.100723 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100566    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bf513a7-048e-42d8-98fb-5b7adbc09011-serviceca\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.101586 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvftl\" (UniqueName: \"kubernetes.io/projected/37a32307-3434-4dff-91d1-8dd3e6c8369e-kube-api-access-nvftl\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.101586 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100689    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825a637-bc1a-4b66-acd1-c0081284ae23-tmp-dir\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.101586 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.100718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-hostroot\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.123305 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.123279    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:31:44.123500 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.123364    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kgtdv"
Oct 30 12:31:44.128505 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.128482    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:31:44.128505 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.128500    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:31:44.128680 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.128496    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7nlfh\""
Oct 30 12:31:44.128680 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.128486    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:31:44.128680 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.128630    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:31:44.128931 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.128912    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5fvrw\""
Oct 30 12:31:44.128931 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.128921    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:31:44.129058 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.128947    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgbt6\""
Oct 30 12:31:44.129058 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129007    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:31:44.129058 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129026    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:31:44.129058 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129037    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:31:44.129223 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129111    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kq49t\""
Oct 30 12:31:44.129223 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129214    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:31:44.129510 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129489    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:31:44.129605 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129583    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:31:44.129728 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129704    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxtxl\""
Oct 30 12:31:44.129827 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129744    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:31:44.129827 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129790    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:31:44.129940 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129893    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:31:44.129978 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129940    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:31:44.130030 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.129998    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:31:44.130218 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.130178    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:31:44.136996 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.136975    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kgtdv"
Oct 30 12:31:44.201867 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.201821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-tmp\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.202068 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.201884    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-registration-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.202068 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.201952    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-device-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.202068 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.201992    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-cnibin\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.202068 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-ovn\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.202246 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202071    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-registration-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.202246 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202153    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-bin\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.202246 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202170    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-device-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.202246 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-kubelet\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.202246 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-daemon-config\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.202246 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202230    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-ovn\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.202473 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202249    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:44.202473 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202281    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-cnibin\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.202473 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.202473 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202331    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-bin\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.202473 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202301    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:31:44.202473 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202444    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.202473 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.202460    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:44.202773 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.202604    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:31:44.702559109 +0000 UTC m=+3.154322253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:44.202773 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202633    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-kubelet\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.202773 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202675    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-env-overrides\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.202773 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202709    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.202773 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-sys-fs\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.202773 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7dm\" (UniqueName: \"kubernetes.io/projected/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kube-api-access-7w7dm\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.203027 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202791    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:44.203027 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-slash\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.203027 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-var-lib-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.203027 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202893    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-log-socket\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.203027 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202918    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-system-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.203027 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202921    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-sys-fs\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.203027 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.202966    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-socket-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.203027 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-systemd\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-script-lib\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztsmb\" (UniqueName: \"kubernetes.io/projected/4825a637-bc1a-4b66-acd1-c0081284ae23-kube-api-access-ztsmb\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-etc-kubernetes\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203164    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc2566d-5249-44ca-be3a-697d3eb4d603-host-slash\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203191    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-daemon-config\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203192    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-etc-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203329    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-system-cni-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203335    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-etc-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-netns\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.203453 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-conf-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203465    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqfh\" (UniqueName: \"kubernetes.io/projected/adf5def0-890c-405c-9706-664675f53547-kube-api-access-9dqfh\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203493    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203519    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cee19c40-c94c-45cd-8deb-16e8a965e053-konnectivity-ca\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203528    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-env-overrides\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4825a637-bc1a-4b66-acd1-c0081284ae23-hosts-file\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203645    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4825a637-bc1a-4b66-acd1-c0081284ae23-hosts-file\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203690    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-system-cni-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203733    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxsh\" (UniqueName: \"kubernetes.io/projected/f2102d05-9eb2-4641-a057-7b02ae7d88bd-kube-api-access-rtxsh\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203757    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-netns\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203786    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-kubernetes\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203801    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-conf-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.203952 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203807    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203983    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-slash\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204058    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-kubernetes\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204102    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-var-lib-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204110    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-systemd\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204159    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-log-socket\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204176    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-socket-dir-parent\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204208    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.203805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-socket-dir-parent\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204233    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-etc-kubernetes\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204297    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-system-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204387    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc2566d-5249-44ca-be3a-697d3eb4d603-host-slash\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.204509 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204502    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-socket-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204555    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-kubelet-config\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204611    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-node-log\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-systemd-units\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204682    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-script-lib\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-multus\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204715    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-kubelet-config\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204755    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-node-log\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204766    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-host\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204771    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-multus\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204780    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-systemd-units\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-etc-tuned\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204824    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-host\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204858    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-systemd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.204888 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204869    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cee19c40-c94c-45cd-8deb-16e8a965e053-konnectivity-ca\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204885    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-systemd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-config\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204952    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.204981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cnibin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205042    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjh8\" (UniqueName: \"kubernetes.io/projected/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-kube-api-access-cmjh8\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-dbus\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205122    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf513a7-048e-42d8-98fb-5b7adbc09011-host\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205176    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cnibin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205251    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-dbus\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205287    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf513a7-048e-42d8-98fb-5b7adbc09011-host\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205335    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cni-binary-copy\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-k8s-cni-cncf-io\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205572    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.205555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205611    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysconfig\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-run\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-netns\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205872    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205945    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-config\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205960    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-multus-certs\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.205986    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75mt\" (UniqueName: \"kubernetes.io/projected/4bc2566d-5249-44ca-be3a-697d3eb4d603-kube-api-access-j75mt\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.206021    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysconfig\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.206033    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-var-lib-kubelet\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.206091    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-var-lib-kubelet\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.206141    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-netns\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.206466 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.206194    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.207177 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.206781    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.207177 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.206844    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-multus-certs\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.207177 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.206628    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.207177 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207130    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cni-binary-copy\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.207177 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207168    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-k8s-cni-cncf-io\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207205    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-run\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-kubelet\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207259    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-netd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207314    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-kubelet\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovn-node-metrics-cert\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207371    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-netd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.207439 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7cq\" (UniqueName: \"kubernetes.io/projected/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-kube-api-access-6w7cq\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-bin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-conf\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-sys\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207558    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8h5\" (UniqueName: \"kubernetes.io/projected/6bf513a7-048e-42d8-98fb-5b7adbc09011-kube-api-access-zx8h5\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207582    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cee19c40-c94c-45cd-8deb-16e8a965e053-agent-certs\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207661    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207685    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bc2566d-5249-44ca-be3a-697d3eb4d603-iptables-alerter-script\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.207736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bf513a7-048e-42d8-98fb-5b7adbc09011-serviceca\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.208210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207760    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvftl\" (UniqueName: \"kubernetes.io/projected/37a32307-3434-4dff-91d1-8dd3e6c8369e-kube-api-access-nvftl\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.208210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207786    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825a637-bc1a-4b66-acd1-c0081284ae23-tmp-dir\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.208210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-hostroot\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.208210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207833    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-modprobe-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.208210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-lib-modules\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.208210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207881    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-tmp\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.208210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.207982    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-bin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.208210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.208179    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-lib-modules\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.208615 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.208549    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-hostroot\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.209207    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-conf\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.209269    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-sys\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.209450    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825a637-bc1a-4b66-acd1-c0081284ae23-tmp-dir\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.209638    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.209657    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-modprobe-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.209857    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bf513a7-048e-42d8-98fb-5b7adbc09011-serviceca\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.209895    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-etc-tuned\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.210075    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.210077    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovn-node-metrics-cert\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.210174    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret podName:a3349844-7e31-4934-8c31-2a208adbf1a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:44.71015423 +0000 UTC m=+3.161917354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret") pod "global-pull-secret-syncer-6h57s" (UID: "a3349844-7e31-4934-8c31-2a208adbf1a5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:44.210437 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.210275    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.211309 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.211283    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bc2566d-5249-44ca-be3a-697d3eb4d603-iptables-alerter-script\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.215604 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.215504    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cee19c40-c94c-45cd-8deb-16e8a965e053-agent-certs\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:44.240124 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.240099    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:44.240124 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.240128    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:44.240325 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.240143    2571 projected.go:196] Error preparing data for projected volume kube-api-access-djhzw for pod openshift-network-diagnostics/network-check-target-cqc2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:44.240325 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.240219    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw podName:a3c865c9-11ec-4356-b33c-b0e4d2e6a401 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:44.740200542 +0000 UTC m=+3.191963686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-djhzw" (UniqueName: "kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw") pod "network-check-target-cqc2z" (UID: "a3c865c9-11ec-4356-b33c-b0e4d2e6a401") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:44.242357 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.242312    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxsh\" (UniqueName: \"kubernetes.io/projected/f2102d05-9eb2-4641-a057-7b02ae7d88bd-kube-api-access-rtxsh\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:44.242631 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.242612    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqfh\" (UniqueName: \"kubernetes.io/projected/adf5def0-890c-405c-9706-664675f53547-kube-api-access-9dqfh\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.247164 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.246923    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztsmb\" (UniqueName: \"kubernetes.io/projected/4825a637-bc1a-4b66-acd1-c0081284ae23-kube-api-access-ztsmb\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.248976 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.248331    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75mt\" (UniqueName: \"kubernetes.io/projected/4bc2566d-5249-44ca-be3a-697d3eb4d603-kube-api-access-j75mt\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.248976 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.248929    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8h5\" (UniqueName: \"kubernetes.io/projected/6bf513a7-048e-42d8-98fb-5b7adbc09011-kube-api-access-zx8h5\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.249754 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.249730    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7cq\" (UniqueName: \"kubernetes.io/projected/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-kube-api-access-6w7cq\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.250935 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.250912    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjh8\" (UniqueName: \"kubernetes.io/projected/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-kube-api-access-cmjh8\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.250935 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.250946    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7dm\" (UniqueName: \"kubernetes.io/projected/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kube-api-access-7w7dm\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.253426 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.253405    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvftl\" (UniqueName: \"kubernetes.io/projected/37a32307-3434-4dff-91d1-8dd3e6c8369e-kube-api-access-nvftl\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.315924 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.315891    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:44.326154 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.325968    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fttst"
Oct 30 12:31:44.329937 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.329900    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee19c40_c94c_45cd_8deb_16e8a965e053.slice/crio-f8715811d525446c4b76e20708251cafeefd1e7a20fd5af1ae5269ac58ec4944 WatchSource:0}: Error finding container f8715811d525446c4b76e20708251cafeefd1e7a20fd5af1ae5269ac58ec4944: Status 404 returned error can't find the container with id f8715811d525446c4b76e20708251cafeefd1e7a20fd5af1ae5269ac58ec4944
Oct 30 12:31:44.337905 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.337880    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:31:44.339713 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.339686    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4825a637_bc1a_4b66_acd1_c0081284ae23.slice/crio-87d78bfd9709d2f7366d2667c96a089f3f7488515a47a35a9b286a37b3811bf4 WatchSource:0}: Error finding container 87d78bfd9709d2f7366d2667c96a089f3f7488515a47a35a9b286a37b3811bf4: Status 404 returned error can't find the container with id 87d78bfd9709d2f7366d2667c96a089f3f7488515a47a35a9b286a37b3811bf4
Oct 30 12:31:44.345118 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.345093    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k4c5j"
Oct 30 12:31:44.350909 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.350849    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf5def0_890c_405c_9706_664675f53547.slice/crio-8ad130858c1e07bc47d4091ad483db49ac3aca7b9e74c479e3d8a2b6c5e19a1e WatchSource:0}: Error finding container 8ad130858c1e07bc47d4091ad483db49ac3aca7b9e74c479e3d8a2b6c5e19a1e: Status 404 returned error can't find the container with id 8ad130858c1e07bc47d4091ad483db49ac3aca7b9e74c479e3d8a2b6c5e19a1e
Oct 30 12:31:44.351877 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.351856    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:31:44.357509 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.357472    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e07537a_35f5_40ad_b6fe_6b0c92717cf4.slice/crio-bbf1d4cb6e81c96ffcc9c384d22760f6393a5d852cb1979b45aac97bf2137320 WatchSource:0}: Error finding container bbf1d4cb6e81c96ffcc9c384d22760f6393a5d852cb1979b45aac97bf2137320: Status 404 returned error can't find the container with id bbf1d4cb6e81c96ffcc9c384d22760f6393a5d852cb1979b45aac97bf2137320
Oct 30 12:31:44.362239 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.362055    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:31:44.364453 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.364405    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf513a7_048e_42d8_98fb_5b7adbc09011.slice/crio-5e9c85b3c5dcf82d1181b64cc6515f91f2ca019d65ee50dd806cc3cdf49f5c58 WatchSource:0}: Error finding container 5e9c85b3c5dcf82d1181b64cc6515f91f2ca019d65ee50dd806cc3cdf49f5c58: Status 404 returned error can't find the container with id 5e9c85b3c5dcf82d1181b64cc6515f91f2ca019d65ee50dd806cc3cdf49f5c58
Oct 30 12:31:44.373222 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.373200    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:31:44.375543 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.375504    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc2566d_5249_44ca_be3a_697d3eb4d603.slice/crio-0593874be2f59400ad0aa1d2854eb6e0d905458c37398f0c1f8c9f41c53221eb WatchSource:0}: Error finding container 0593874be2f59400ad0aa1d2854eb6e0d905458c37398f0c1f8c9f41c53221eb: Status 404 returned error can't find the container with id 0593874be2f59400ad0aa1d2854eb6e0d905458c37398f0c1f8c9f41c53221eb
Oct 30 12:31:44.392038 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.391994    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f03eb8_15da_4d44_bc9c_8e043931b8ee.slice/crio-99a5403e2faa3a70828bc4291407dcc76d07127f0f7965a89f5baa72ca968060 WatchSource:0}: Error finding container 99a5403e2faa3a70828bc4291407dcc76d07127f0f7965a89f5baa72ca968060: Status 404 returned error can't find the container with id 99a5403e2faa3a70828bc4291407dcc76d07127f0f7965a89f5baa72ca968060
Oct 30 12:31:44.397856 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.397834    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:31:44.408943 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.408899    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:31:44.409158 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.409134    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9729abbf_37cc_43cb_bbf9_8c08a18eb37a.slice/crio-adf93a4a12fbee1e7d7304122d390fe16a8007a6a1cb09c9d8b0a6d23b0d04a2 WatchSource:0}: Error finding container adf93a4a12fbee1e7d7304122d390fe16a8007a6a1cb09c9d8b0a6d23b0d04a2: Status 404 returned error can't find the container with id adf93a4a12fbee1e7d7304122d390fe16a8007a6a1cb09c9d8b0a6d23b0d04a2
Oct 30 12:31:44.441919 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:31:44.441893    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a32307_3434_4dff_91d1_8dd3e6c8369e.slice/crio-b0766ac3619e522c5a0703fd965d57fc9d07d4e8a6e6e306063eb659bae81e02 WatchSource:0}: Error finding container b0766ac3619e522c5a0703fd965d57fc9d07d4e8a6e6e306063eb659bae81e02: Status 404 returned error can't find the container with id b0766ac3619e522c5a0703fd965d57fc9d07d4e8a6e6e306063eb659bae81e02
Oct 30 12:31:44.711670 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.711641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:44.711759 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.711685    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:44.711804 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.711780    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:44.711839 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.711800    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:44.711839 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.711832    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:31:45.711818158 +0000 UTC m=+4.163581291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:44.711911 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.711874    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret podName:a3349844-7e31-4934-8c31-2a208adbf1a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:45.711859841 +0000 UTC m=+4.163622959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret") pod "global-pull-secret-syncer-6h57s" (UID: "a3349844-7e31-4934-8c31-2a208adbf1a5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:44.812920 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:44.812664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:44.813086 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.813057    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:44.813086 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.813076    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:44.813086 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.813088    2571 projected.go:196] Error preparing data for projected volume kube-api-access-djhzw for pod openshift-network-diagnostics/network-check-target-cqc2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:44.813251 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:44.813196    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw podName:a3c865c9-11ec-4356-b33c-b0e4d2e6a401 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:45.813177432 +0000 UTC m=+4.264940565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhzw" (UniqueName: "kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw") pod "network-check-target-cqc2z" (UID: "a3c865c9-11ec-4356-b33c-b0e4d2e6a401") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:45.122771 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.121591    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:45.122771 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.121787    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:45.122771 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.121944    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:45.122771 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.122036    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:45.122771 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.122111    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:45.122771 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.122203    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:45.138905 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.138855    2571 generic.go:356] "Generic (PLEG): container finished" podID="3a7fe78ff9a981c06dfba04265b1aa94" containerID="8e31d351c9871cbb7780a40cd7f0e934d63e1c562de7ad87eb34348b37538b85" exitCode=0
Oct 30 12:31:45.139101 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.138970    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal" event={"ID":"3a7fe78ff9a981c06dfba04265b1aa94","Type":"ContainerDied","Data":"8e31d351c9871cbb7780a40cd7f0e934d63e1c562de7ad87eb34348b37538b85"}
Oct 30 12:31:45.140577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.140467    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:44 +0000 UTC" deadline="2028-07-08 14:45:48.782624324 +0000 UTC"
Oct 30 12:31:45.140577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.140518    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14810h14m3.642110877s"
Oct 30 12:31:45.144975 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.144457    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal" event={"ID":"e9b5f11662f569a9096e79917d2f933f","Type":"ContainerStarted","Data":"3d21452362ce951a37e27b401b77eb14bb87a1ca09b0c426dffeec7ed19195cf"}
Oct 30 12:31:45.154934 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.154709    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" event={"ID":"37a32307-3434-4dff-91d1-8dd3e6c8369e","Type":"ContainerStarted","Data":"b0766ac3619e522c5a0703fd965d57fc9d07d4e8a6e6e306063eb659bae81e02"}
Oct 30 12:31:45.159538 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.159424    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn" event={"ID":"f1f03eb8-15da-4d44-bc9c-8e043931b8ee","Type":"ContainerStarted","Data":"99a5403e2faa3a70828bc4291407dcc76d07127f0f7965a89f5baa72ca968060"}
Oct 30 12:31:45.162465 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.162436    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qmdgt" event={"ID":"4bc2566d-5249-44ca-be3a-697d3eb4d603","Type":"ContainerStarted","Data":"0593874be2f59400ad0aa1d2854eb6e0d905458c37398f0c1f8c9f41c53221eb"}
Oct 30 12:31:45.171487 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.171398    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r8lf8" event={"ID":"6bf513a7-048e-42d8-98fb-5b7adbc09011","Type":"ContainerStarted","Data":"5e9c85b3c5dcf82d1181b64cc6515f91f2ca019d65ee50dd806cc3cdf49f5c58"}
Oct 30 12:31:45.178820 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.178708    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k4c5j" event={"ID":"8e07537a-35f5-40ad-b6fe-6b0c92717cf4","Type":"ContainerStarted","Data":"bbf1d4cb6e81c96ffcc9c384d22760f6393a5d852cb1979b45aac97bf2137320"}
Oct 30 12:31:45.181047 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.180924    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"adf93a4a12fbee1e7d7304122d390fe16a8007a6a1cb09c9d8b0a6d23b0d04a2"}
Oct 30 12:31:45.191469 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.191337    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lgqrc" event={"ID":"adf5def0-890c-405c-9706-664675f53547","Type":"ContainerStarted","Data":"8ad130858c1e07bc47d4091ad483db49ac3aca7b9e74c479e3d8a2b6c5e19a1e"}
Oct 30 12:31:45.199126 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.199073    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fttst" event={"ID":"4825a637-bc1a-4b66-acd1-c0081284ae23","Type":"ContainerStarted","Data":"87d78bfd9709d2f7366d2667c96a089f3f7488515a47a35a9b286a37b3811bf4"}
Oct 30 12:31:45.210016 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.209919    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xz8gl" event={"ID":"cee19c40-c94c-45cd-8deb-16e8a965e053","Type":"ContainerStarted","Data":"f8715811d525446c4b76e20708251cafeefd1e7a20fd5af1ae5269ac58ec4944"}
Oct 30 12:31:45.720795 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.720705    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:45.721023 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.720820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:45.721023 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.720969    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:45.721134 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.721039    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret podName:a3349844-7e31-4934-8c31-2a208adbf1a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:47.721017248 +0000 UTC m=+6.172780393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret") pod "global-pull-secret-syncer-6h57s" (UID: "a3349844-7e31-4934-8c31-2a208adbf1a5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:45.721539 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.721516    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:45.721671 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.721583    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:31:47.72156427 +0000 UTC m=+6.173327405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:45.821465 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:45.821415    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:45.821676 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.821653    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:45.821758 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.821679    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:45.821758 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.821694    2571 projected.go:196] Error preparing data for projected volume kube-api-access-djhzw for pod openshift-network-diagnostics/network-check-target-cqc2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:45.821868 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:45.821759    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw podName:a3c865c9-11ec-4356-b33c-b0e4d2e6a401 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:47.821739551 +0000 UTC m=+6.273502904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhzw" (UniqueName: "kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw") pod "network-check-target-cqc2z" (UID: "a3c865c9-11ec-4356-b33c-b0e4d2e6a401") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:46.019555 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:46.019351    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:31:46.141177 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:46.141102    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:44 +0000 UTC" deadline="2028-06-09 20:13:41.776934502 +0000 UTC"
Oct 30 12:31:46.141177 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:46.141141    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14119h41m55.635798146s"
Oct 30 12:31:46.236908 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:46.236845    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal" event={"ID":"3a7fe78ff9a981c06dfba04265b1aa94","Type":"ContainerStarted","Data":"f2553d278b8a0cff7da05243849c1372e86081eb26c7551720ea53c581841488"}
Oct 30 12:31:46.277530 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:46.277397    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal" podStartSLOduration=2.277358628 podStartE2EDuration="2.277358628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:45.219229818 +0000 UTC m=+3.670992963" watchObservedRunningTime="2026-10-30 12:31:46.277358628 +0000 UTC m=+4.729121773"
Oct 30 12:31:47.121585 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:47.121545    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:47.121786 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.121715    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:47.122143 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:47.122119    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:47.122264 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.122238    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:47.122325 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:47.122298    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:47.122417 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.122397    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:47.737020 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:47.736935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:47.737020 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:47.736996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:47.737617 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.737145    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:47.737617 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.737206    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:31:51.737190778 +0000 UTC m=+10.188953914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:47.738240 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.738171    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:47.738521 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.738507    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret podName:a3349844-7e31-4934-8c31-2a208adbf1a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:51.738473793 +0000 UTC m=+10.190236927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret") pod "global-pull-secret-syncer-6h57s" (UID: "a3349844-7e31-4934-8c31-2a208adbf1a5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:47.838951 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:47.838405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:47.838951 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.838575    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:47.838951 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.838595    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:47.838951 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.838605    2571 projected.go:196] Error preparing data for projected volume kube-api-access-djhzw for pod openshift-network-diagnostics/network-check-target-cqc2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:47.838951 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:47.838663    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw podName:a3c865c9-11ec-4356-b33c-b0e4d2e6a401 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:51.838645283 +0000 UTC m=+10.290408427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhzw" (UniqueName: "kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw") pod "network-check-target-cqc2z" (UID: "a3c865c9-11ec-4356-b33c-b0e4d2e6a401") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:48.246262 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:48.245113    2571 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="d505ee458da66ce5037247f6b83b0eef2698d0677100638e79a18e5ce9107faa" exitCode=0
Oct 30 12:31:48.246262 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:48.245160    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" event={"ID":"37a32307-3434-4dff-91d1-8dd3e6c8369e","Type":"ContainerDied","Data":"d505ee458da66ce5037247f6b83b0eef2698d0677100638e79a18e5ce9107faa"}
Oct 30 12:31:48.316464 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:48.316391    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal" podStartSLOduration=4.316348469 podStartE2EDuration="4.316348469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:46.277747739 +0000 UTC m=+4.729510886" watchObservedRunningTime="2026-10-30 12:31:48.316348469 +0000 UTC m=+6.768111611"
Oct 30 12:31:49.121701 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:49.121663    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:49.122214 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:49.121801    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:49.122214 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:49.121904    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:49.122214 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:49.122083    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:49.122214 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:49.122155    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:49.122445 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:49.122244    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:49.250176 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:49.250132    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xz8gl" event={"ID":"cee19c40-c94c-45cd-8deb-16e8a965e053","Type":"ContainerStarted","Data":"4906d19ad7ab1ff837f2cbcb094f72ecf88ae26da58fbb6a1cc083d4bec09be0"}
Oct 30 12:31:50.256841 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:50.256233    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r8lf8" event={"ID":"6bf513a7-048e-42d8-98fb-5b7adbc09011","Type":"ContainerStarted","Data":"079d53bcfbe9b7ae4edaee67039401dc8de7011fc24fdad59109a459209b290f"}
Oct 30 12:31:50.294111 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:50.294049    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xz8gl" podStartSLOduration=4.387053672 podStartE2EDuration="8.294029092s" totalImagesPullingTime="3.90697542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:49.292692577 +0000 UTC m=+7.744455723" watchObservedRunningTime="2026-10-30 12:31:50.294029092 +0000 UTC m=+8.745792235"
Oct 30 12:31:51.120919 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:51.120885    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:51.121099 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.121012    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:51.121413 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:51.121387    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:51.121534 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.121471    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:51.121534 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:51.121504    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:51.121633 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.121543    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:51.267687 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:51.267405    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn" event={"ID":"f1f03eb8-15da-4d44-bc9c-8e043931b8ee","Type":"ContainerStarted","Data":"7af50560d50f6427d11d31275bd6af56b22bb8423c8cea021b58188d2a3e6014"}
Oct 30 12:31:51.775226 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:51.775181    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:51.775397 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:51.775277    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:51.775477 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.775460    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:51.775551 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.775539    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:31:59.775520977 +0000 UTC m=+18.227284111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:51.775956 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.775938    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:51.776017 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.775996    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret podName:a3349844-7e31-4934-8c31-2a208adbf1a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:59.775981275 +0000 UTC m=+18.227744410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret") pod "global-pull-secret-syncer-6h57s" (UID: "a3349844-7e31-4934-8c31-2a208adbf1a5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:51.862934 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:51.862906    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:31:51.875936 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:51.875906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:51.876098 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.876078    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:51.876174 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.876105    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:51.876174 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.876119    2571 projected.go:196] Error preparing data for projected volume kube-api-access-djhzw for pod openshift-network-diagnostics/network-check-target-cqc2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:51.876174 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:51.876172    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw podName:a3c865c9-11ec-4356-b33c-b0e4d2e6a401 nodeName:}" failed. No retries permitted until 2026-10-30 12:31:59.876152923 +0000 UTC m=+18.327916053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhzw" (UniqueName: "kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw") pod "network-check-target-cqc2z" (UID: "a3c865c9-11ec-4356-b33c-b0e4d2e6a401") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:52.056763 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:52.055862    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:31:51.862930451Z","UUID":"6ff8b61d-6bf6-4502-b681-7cfe9e4f4b95","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:31:52.061921 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:52.061408    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:31:52.061921 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:52.061437    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:31:52.268830 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:52.268750    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn" event={"ID":"f1f03eb8-15da-4d44-bc9c-8e043931b8ee","Type":"ContainerStarted","Data":"3357e85e18f9c74d5ba5b106054fb0b0bd4a6fefb82e3648f75f67b6d34d934f"}
Oct 30 12:31:53.121177 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.120989    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:53.121307 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.121039    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:53.121366 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:53.121323    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:53.121366 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.121051    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:53.121469 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:53.121399    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:53.121508 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:53.121469    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:53.274010 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.273976    2571 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="c067dd615d7ba695901719e05341966aefbe96a28afaf000da4ab3b4418d3d85" exitCode=0
Oct 30 12:31:53.274462 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.274069    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" event={"ID":"37a32307-3434-4dff-91d1-8dd3e6c8369e","Type":"ContainerDied","Data":"c067dd615d7ba695901719e05341966aefbe96a28afaf000da4ab3b4418d3d85"}
Oct 30 12:31:53.278313 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.277668    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn" event={"ID":"f1f03eb8-15da-4d44-bc9c-8e043931b8ee","Type":"ContainerStarted","Data":"ca86d8ec31477281f3687101eec238d2380fc2b3c0b4e152409eeabf3701c4dd"}
Oct 30 12:31:53.288433 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.288401    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k4c5j" event={"ID":"8e07537a-35f5-40ad-b6fe-6b0c92717cf4","Type":"ContainerStarted","Data":"b4099878cfd10876f20481716286af83c0b35dc917b670d4dcaa158a07d174e3"}
Oct 30 12:31:53.289946 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.289925    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fttst" event={"ID":"4825a637-bc1a-4b66-acd1-c0081284ae23","Type":"ContainerStarted","Data":"1612b72ea2ceaf961eb7b3e91360ec5725aace81eb287e8992627598cd5550b2"}
Oct 30 12:31:53.347658 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.347603    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r8lf8" podStartSLOduration=6.091961204 podStartE2EDuration="11.347587238s" totalImagesPullingTime="5.255626034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:50.295081475 +0000 UTC m=+8.746844618" watchObservedRunningTime="2026-10-30 12:31:53.347587238 +0000 UTC m=+11.799350378"
Oct 30 12:31:53.374711 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.374658    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-k4c5j" podStartSLOduration=3.403988595 podStartE2EDuration="11.374645468s" totalImagesPullingTime="7.970656873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:53.374206217 +0000 UTC m=+11.825969357" watchObservedRunningTime="2026-10-30 12:31:53.374645468 +0000 UTC m=+11.826408607"
Oct 30 12:31:53.455381 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.455319    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn" podStartSLOduration=3.126226543 podStartE2EDuration="11.455301968s" totalImagesPullingTime="8.329075425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:53.413829821 +0000 UTC m=+11.865592960" watchObservedRunningTime="2026-10-30 12:31:53.455301968 +0000 UTC m=+11.907065104"
Oct 30 12:31:53.577807 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.577776    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:53.578402 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.578378    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:53.615164 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:53.615102    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-fttst" podStartSLOduration=3.644705374 podStartE2EDuration="11.615087651s" totalImagesPullingTime="7.970382277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:53.456169835 +0000 UTC m=+11.907932975" watchObservedRunningTime="2026-10-30 12:31:53.615087651 +0000 UTC m=+12.066850821"
Oct 30 12:31:54.295406 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:54.295321    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" event={"ID":"37a32307-3434-4dff-91d1-8dd3e6c8369e","Type":"ContainerDied","Data":"4a925a5a1e12cfda231bacbe423ab29f245c7dc9235bbe65d9bb98502c2c5ff1"}
Oct 30 12:31:54.295934 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:54.295004    2571 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="4a925a5a1e12cfda231bacbe423ab29f245c7dc9235bbe65d9bb98502c2c5ff1" exitCode=0
Oct 30 12:31:54.298026 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:54.297997    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qmdgt" event={"ID":"4bc2566d-5249-44ca-be3a-697d3eb4d603","Type":"ContainerStarted","Data":"09a5198512203c00b7d88f8c1c5b09bbcc775f06168d208836bcde744babbdd9"}
Oct 30 12:31:54.298210 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:54.298150    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:54.298831 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:54.298811    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:31:54.415910 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:54.415854    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qmdgt" podStartSLOduration=4.242090368 podStartE2EDuration="12.415839676s" totalImagesPullingTime="8.173749308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:54.415715257 +0000 UTC m=+12.867478404" watchObservedRunningTime="2026-10-30 12:31:54.415839676 +0000 UTC m=+12.867602815"
Oct 30 12:31:55.121767 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:55.121732    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:55.121767 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:55.121762    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:55.122004 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:55.121736    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:55.122004 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:55.121845    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:55.122004 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:55.121916    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:55.122913 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:55.122888    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:55.301857 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:55.301819    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lgqrc" event={"ID":"adf5def0-890c-405c-9706-664675f53547","Type":"ContainerStarted","Data":"c274ab11ff9bea417c72afd68e8bfeb20d8ebdbc691437f90fbab91eae29b784"}
Oct 30 12:31:55.304445 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:55.304418    2571 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="bb2ddbba56eff50eadb473ec6d021bcf015bad16ce4b8643d6d5b10aef4163d8" exitCode=0
Oct 30 12:31:55.304562 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:55.304449    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" event={"ID":"37a32307-3434-4dff-91d1-8dd3e6c8369e","Type":"ContainerDied","Data":"bb2ddbba56eff50eadb473ec6d021bcf015bad16ce4b8643d6d5b10aef4163d8"}
Oct 30 12:31:55.340403 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:55.340358    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lgqrc" podStartSLOduration=3.168379926 podStartE2EDuration="13.340345885s" totalImagesPullingTime="10.171965959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:55.340193115 +0000 UTC m=+13.791956253" watchObservedRunningTime="2026-10-30 12:31:55.340345885 +0000 UTC m=+13.792109024"
Oct 30 12:31:57.120835 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:57.120670    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:57.121255 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:57.120670    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:57.121255 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:57.120932    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:57.121255 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:57.121010    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:57.121255 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:57.120670    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:57.121255 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:57.121096    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:57.312962 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:57.312922    2571 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="466eb53c5ff85ecd5cbc3b64732bcbcc50575d629db08b89b6a2ce67d28614ec" exitCode=0
Oct 30 12:31:57.313127 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:57.312984    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" event={"ID":"37a32307-3434-4dff-91d1-8dd3e6c8369e","Type":"ContainerDied","Data":"466eb53c5ff85ecd5cbc3b64732bcbcc50575d629db08b89b6a2ce67d28614ec"}
Oct 30 12:31:58.318951 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:58.318918    2571 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="ac3879fe50009fbf8d3cd395345219667d8b1f4911d62846062dc8aa86a57695" exitCode=0
Oct 30 12:31:58.319432 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:58.318979    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" event={"ID":"37a32307-3434-4dff-91d1-8dd3e6c8369e","Type":"ContainerDied","Data":"ac3879fe50009fbf8d3cd395345219667d8b1f4911d62846062dc8aa86a57695"}
Oct 30 12:31:59.121338 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:59.121301    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:59.121499 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:59.121394    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:59.121499 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:59.121403    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:59.121499 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.121398    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:31:59.121628 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.121485    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:31:59.121628 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.121545    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:31:59.324033 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:59.323998    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" event={"ID":"37a32307-3434-4dff-91d1-8dd3e6c8369e","Type":"ContainerStarted","Data":"5944138e94dbba4cb4445d9aae2400c56dafdd2dd866224b5490386ae072541c"}
Oct 30 12:31:59.390450 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:59.390354    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fpqzd" podStartSLOduration=6.633123121 podStartE2EDuration="17.390340777s" totalImagesPullingTime="10.757217656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:59.377771283 +0000 UTC m=+17.829534452" watchObservedRunningTime="2026-10-30 12:31:59.390340777 +0000 UTC m=+17.842103915"
Oct 30 12:31:59.836330 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:59.836135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:31:59.836490 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:59.836358    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:31:59.836490 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.836298    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:59.836490 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.836450    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret podName:a3349844-7e31-4934-8c31-2a208adbf1a5 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:15.83643473 +0000 UTC m=+34.288197874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret") pod "global-pull-secret-syncer-6h57s" (UID: "a3349844-7e31-4934-8c31-2a208adbf1a5") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:31:59.836490 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.836481    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:59.836659 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.836552    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:32:15.836534995 +0000 UTC m=+34.288298127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:31:59.937077 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:31:59.937045    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:31:59.937221 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.937183    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:31:59.937221 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.937201    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:31:59.937221 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.937212    2571 projected.go:196] Error preparing data for projected volume kube-api-access-djhzw for pod openshift-network-diagnostics/network-check-target-cqc2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:31:59.937423 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:31:59.937272    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw podName:a3c865c9-11ec-4356-b33c-b0e4d2e6a401 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:15.937255187 +0000 UTC m=+34.389018317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhzw" (UniqueName: "kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw") pod "network-check-target-cqc2z" (UID: "a3c865c9-11ec-4356-b33c-b0e4d2e6a401") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:32:00.329158 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:00.329089    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"a6bcd99598ed50a5271e7615c992c45423f91dca20870a5f66a2075cb3cdd8e3"}
Oct 30 12:32:00.329158 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:00.329155    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"1b511115d36f7086291fba360cce8978e0e3a08c88254cb1e5e8c55d32bc68b2"}
Oct 30 12:32:00.329551 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:00.329184    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"e87126b3e7814f81f49852c08700edf239153f3bdc73d9563f5d705cf7e2d5db"}
Oct 30 12:32:00.329551 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:00.329212    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"94e547a394f31d5ee8e687b71b73e8e77c90ae9e07da3d52139a9d724d3cfcbb"}
Oct 30 12:32:00.329551 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:00.329240    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"3ac22bf34e69dd631d776438cbdda9e4d8b0660ba32f039ad3f6606262b90a6e"}
Oct 30 12:32:00.329551 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:00.329266    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"b114152a73f69465b7e104201c88f2313c8452ade394dbff0e439d441b97eead"}
Oct 30 12:32:01.121424 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:01.121390    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:01.121609 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:01.121479    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:01.121609 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:01.121497    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:01.121609 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:01.121586    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:32:01.121755 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:01.121696    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:32:01.121793 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:01.121760    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:32:02.338573 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:02.338537    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"addb60f09d7bd07158d16b00bc74fd5c6455daab553e6bd259116391a7fc7485"}
Oct 30 12:32:03.121377 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:03.121347    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:03.121556 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:03.121347    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:03.121556 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:03.121459    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:32:03.121556 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:03.121516    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:32:03.121556 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:03.121358    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:03.121693 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:03.121595    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:32:04.346012 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:04.345981    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" event={"ID":"9729abbf-37cc-43cb-bbf9-8c08a18eb37a","Type":"ContainerStarted","Data":"2112fdbfde03a7e4f8399c6028870aed20d5fe9c72ac22e6f5630b68792eea69"}
Oct 30 12:32:04.346619 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:04.346223    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:04.362017 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:04.361993    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:04.444175 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:04.444131    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms" podStartSLOduration=7.439950271 podStartE2EDuration="22.444117038s" totalImagesPullingTime="15.004166767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:04.443580763 +0000 UTC m=+22.895343903" watchObservedRunningTime="2026-10-30 12:32:04.444117038 +0000 UTC m=+22.895880267"
Oct 30 12:32:05.121527 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:05.121270    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:05.121796 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:05.121300    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:05.121796 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:05.121615    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:32:05.121796 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:05.121713    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:32:05.121796 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:05.121300    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:05.121959 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:05.121842    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:32:05.347945 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:05.347918    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:05.348336 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:05.347981    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:05.363224 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:05.363198    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:06.063171 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:06.063089    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6h57s"]
Oct 30 12:32:06.063389 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:06.063253    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:06.063389 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:06.063371    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:32:06.063903 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:06.063861    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfgnl"]
Oct 30 12:32:06.064042 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:06.064013    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:06.064182 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:06.064149    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:32:06.064425 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:06.064406    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cqc2z"]
Oct 30 12:32:06.064508 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:06.064496    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:06.064589 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:06.064572    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:32:08.120940 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:08.120904    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:08.121420 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:08.120904    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:08.121420 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:08.121009    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:32:08.121420 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:08.120908    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:08.121420 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:08.121086    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:32:08.121420 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:08.121164    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:32:10.121331 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:10.121239    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:10.121331 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:10.121264    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:10.121750 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:10.121366    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6h57s" podUID="a3349844-7e31-4934-8c31-2a208adbf1a5"
Oct 30 12:32:10.121750 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:10.121447    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kfgnl" podUID="f2102d05-9eb2-4641-a057-7b02ae7d88bd"
Oct 30 12:32:10.121750 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:10.121498    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:10.121750 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:10.121574    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cqc2z" podUID="a3c865c9-11ec-4356-b33c-b0e4d2e6a401"
Oct 30 12:32:11.836064 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.836035    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeReady"
Oct 30 12:32:11.836515 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.836184    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 12:32:11.911972 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.911937    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cjg4w"]
Oct 30 12:32:11.914827 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.914813    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gnl9r"]
Oct 30 12:32:11.914966 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.914946    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:11.917574 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.917557    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:11.940187 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.940163    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cjg4w"]
Oct 30 12:32:11.941404 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.941388    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fj9wk"]
Oct 30 12:32:11.943983 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.943965    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:32:11.944097 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.944070    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:11.951222 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.951166    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:32:11.951222 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.951182    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rclv\""
Oct 30 12:32:11.951222 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.951194    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:32:11.951673 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.951633    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:32:11.951673 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.951660    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:32:11.951802 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.951716    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:32:11.951802 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.951731    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:32:11.951802 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.951717    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bklll\""
Oct 30 12:32:11.952775 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.952760    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnl9r"]
Oct 30 12:32:11.962679 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.962663    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:32:11.962942 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.962928    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5lgc6\""
Oct 30 12:32:11.968531 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:11.968508    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fj9wk"]
Oct 30 12:32:12.020836 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.020771    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:32:12.023033 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-config-volume\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.023099 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7424da73-f41f-433c-9bae-1402110b3e6d-cert\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:12.023154 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxpk\" (UniqueName: \"kubernetes.io/projected/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-kube-api-access-ksxpk\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.023238 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f35180a3-5aa1-4885-a824-ea3f40d6546a-crio-socket\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.023278 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f35180a3-5aa1-4885-a824-ea3f40d6546a-data-volume\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.023332 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qssl\" (UniqueName: \"kubernetes.io/projected/7424da73-f41f-433c-9bae-1402110b3e6d-kube-api-access-9qssl\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:12.023374 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.023374 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023348    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.023374 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023364    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nznhb\" (UniqueName: \"kubernetes.io/projected/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-api-access-nznhb\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.023482 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023419    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-tmp-dir\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.023482 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.023456    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.121820 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.121777    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:12.121997 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.121878    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:12.122068 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.121996    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:12.123878 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.123864    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-tmp-dir\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.124013 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.123884    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.124013 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.123907    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-config-volume\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.124013 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.123946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7424da73-f41f-433c-9bae-1402110b3e6d-cert\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:12.124190 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.124022    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:32:12.124190 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxpk\" (UniqueName: \"kubernetes.io/projected/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-kube-api-access-ksxpk\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.124190 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.124094    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.624075822 +0000 UTC m=+31.075838956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : configmap "kube-rbac-proxy" not found
Oct 30 12:32:12.124190 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f35180a3-5aa1-4885-a824-ea3f40d6546a-crio-socket\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.124190 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124148    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f35180a3-5aa1-4885-a824-ea3f40d6546a-data-volume\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.124190 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124171    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-tmp-dir\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.124190 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qssl\" (UniqueName: \"kubernetes.io/projected/7424da73-f41f-433c-9bae-1402110b3e6d-kube-api-access-9qssl\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:12.124577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.124577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124236    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.124577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nznhb\" (UniqueName: \"kubernetes.io/projected/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-api-access-nznhb\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.124577 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124491    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f35180a3-5aa1-4885-a824-ea3f40d6546a-crio-socket\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.124577 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.124526    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:12.124577 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.124539    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:12.124577 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.124580    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls podName:7208585d-b1f1-4d3f-9348-2e4abaaa8e26 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.624562794 +0000 UTC m=+31.076325927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls") pod "dns-default-fj9wk" (UID: "7208585d-b1f1-4d3f-9348-2e4abaaa8e26") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:12.124932 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.124599    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:12.624589145 +0000 UTC m=+31.076352271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:12.124932 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124693    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f35180a3-5aa1-4885-a824-ea3f40d6546a-data-volume\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.124932 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.124777    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-config-volume\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.128588 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.128564    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7424da73-f41f-433c-9bae-1402110b3e6d-cert\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:12.138690 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.138667    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s7t7g"]
Oct 30 12:32:12.142426 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.142405    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:32:12.142426 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.142420    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:32:12.142591 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.142420    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:32:12.142591 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.142438    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wqghl\""
Oct 30 12:32:12.142591 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.142493    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.160050 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.160031    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qssl\" (UniqueName: \"kubernetes.io/projected/7424da73-f41f-433c-9bae-1402110b3e6d-kube-api-access-9qssl\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:12.160138 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.160034    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nznhb\" (UniqueName: \"kubernetes.io/projected/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-api-access-nznhb\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.160138 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.160073    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxpk\" (UniqueName: \"kubernetes.io/projected/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-kube-api-access-ksxpk\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.166227 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.166206    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s7t7g"]
Oct 30 12:32:12.225126 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.225098    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4xr\" (UniqueName: \"kubernetes.io/projected/963ec62f-272a-414d-9542-92273189e836-kube-api-access-wv4xr\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.225277 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.225163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/963ec62f-272a-414d-9542-92273189e836-signing-cabundle\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.225277 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.225227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/963ec62f-272a-414d-9542-92273189e836-signing-key\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.227096 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.227080    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:12.240935 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.240915    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 12:32:12.241052 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.240975    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 12:32:12.241052 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.240981    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 12:32:12.241052 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.241023    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:32:12.241052 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.241029    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zprfc\""
Oct 30 12:32:12.241052 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.241081    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-69dkl\""
Oct 30 12:32:12.241312 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.241171    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 12:32:12.241474 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.241457    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sr4bw\""
Oct 30 12:32:12.326301 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.326254    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4xr\" (UniqueName: \"kubernetes.io/projected/963ec62f-272a-414d-9542-92273189e836-kube-api-access-wv4xr\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.326447 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.326335    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/963ec62f-272a-414d-9542-92273189e836-signing-cabundle\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.326447 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.326364    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/963ec62f-272a-414d-9542-92273189e836-signing-key\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.327068 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.327004    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/963ec62f-272a-414d-9542-92273189e836-signing-cabundle\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.328758 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.328736    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/963ec62f-272a-414d-9542-92273189e836-signing-key\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.351058 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.351033    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4xr\" (UniqueName: \"kubernetes.io/projected/963ec62f-272a-414d-9542-92273189e836-kube-api-access-wv4xr\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.458266 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.458241    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:12.540314 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.540148    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cjg4w"]
Oct 30 12:32:12.543063 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:32:12.543034    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7424da73_f41f_433c_9bae_1402110b3e6d.slice/crio-2dcebe8bb23d1172a1ccbb0b8c208c7eaa2248edaf7719ef15e1d52987dab2d4 WatchSource:0}: Error finding container 2dcebe8bb23d1172a1ccbb0b8c208c7eaa2248edaf7719ef15e1d52987dab2d4: Status 404 returned error can't find the container with id 2dcebe8bb23d1172a1ccbb0b8c208c7eaa2248edaf7719ef15e1d52987dab2d4
Oct 30 12:32:12.629736 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.629708    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:12.629871 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.629740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.629871 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.629769    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:12.629871 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.629836    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 12:32:12.629871 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.629867    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:12.630060 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.629872    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:12.630060 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.629892    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:13.629878571 +0000 UTC m=+32.081641688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : configmap "kube-rbac-proxy" not found
Oct 30 12:32:12.630060 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.629913    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:13.629900728 +0000 UTC m=+32.081663880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:12.630060 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:12.629928    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls podName:7208585d-b1f1-4d3f-9348-2e4abaaa8e26 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:13.629921753 +0000 UTC m=+32.081684870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls") pod "dns-default-fj9wk" (UID: "7208585d-b1f1-4d3f-9348-2e4abaaa8e26") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:12.777312 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:12.777259    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s7t7g"]
Oct 30 12:32:12.779530 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:32:12.779500    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963ec62f_272a_414d_9542_92273189e836.slice/crio-12162cd4b5e123cd344dfe97b5c3f483030c25c10058d97848ab57de8a38d0b4 WatchSource:0}: Error finding container 12162cd4b5e123cd344dfe97b5c3f483030c25c10058d97848ab57de8a38d0b4: Status 404 returned error can't find the container with id 12162cd4b5e123cd344dfe97b5c3f483030c25c10058d97848ab57de8a38d0b4
Oct 30 12:32:13.366921 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:13.366872    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g" event={"ID":"963ec62f-272a-414d-9542-92273189e836","Type":"ContainerStarted","Data":"12162cd4b5e123cd344dfe97b5c3f483030c25c10058d97848ab57de8a38d0b4"}
Oct 30 12:32:13.368408 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:13.368379    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cjg4w" event={"ID":"7424da73-f41f-433c-9bae-1402110b3e6d","Type":"ContainerStarted","Data":"2dcebe8bb23d1172a1ccbb0b8c208c7eaa2248edaf7719ef15e1d52987dab2d4"}
Oct 30 12:32:13.640083 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:13.640036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:13.640083 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:13.640089    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:13.640386 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:13.640123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:13.640386 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:13.640210    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:13.640386 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:13.640258    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:13.640386 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:13.640352    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls podName:7208585d-b1f1-4d3f-9348-2e4abaaa8e26 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:15.640329059 +0000 UTC m=+34.092092181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls") pod "dns-default-fj9wk" (UID: "7208585d-b1f1-4d3f-9348-2e4abaaa8e26") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:13.640386 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:13.640375    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:15.640365044 +0000 UTC m=+34.092128169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:13.640760 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:13.640711    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:15.203442 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.203413    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fttst_4825a637-bc1a-4b66-acd1-c0081284ae23/dns-node-resolver/0.log"
Oct 30 12:32:15.377208 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.377176    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cjg4w" event={"ID":"7424da73-f41f-433c-9bae-1402110b3e6d","Type":"ContainerStarted","Data":"fc1baed1ae2071ebd9f9a65771ef4105d5309f9a7c7388cd7fc6d0d5fe0c26f0"}
Oct 30 12:32:15.657800 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.657727    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:15.657800 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.657758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:15.657973 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:15.657858    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:15.657973 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:15.657871    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:15.657973 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:15.657926    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:19.657909041 +0000 UTC m=+38.109672181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:15.657973 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:15.657939    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls podName:7208585d-b1f1-4d3f-9348-2e4abaaa8e26 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:19.657933261 +0000 UTC m=+38.109696377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls") pod "dns-default-fj9wk" (UID: "7208585d-b1f1-4d3f-9348-2e4abaaa8e26") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:15.798339 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.798317    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r8lf8_6bf513a7-048e-42d8-98fb-5b7adbc09011/node-ca/0.log"
Oct 30 12:32:15.859125 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.859106    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:15.859220 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.859137    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:15.859306 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:15.859232    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:15.859306 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:15.859276    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:32:47.859263116 +0000 UTC m=+66.311026233 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : secret "metrics-daemon-secret" not found
Oct 30 12:32:15.861668 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.861644    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:15.959897 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.959874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:15.962522 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:15.962496    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:16.041925 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.041897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:16.052167 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.052144    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:16.346232 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.346133    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cjg4w" podStartSLOduration=2.659351118 podStartE2EDuration="5.346114471s" totalImagesPullingTime="2.686763353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:15.421388504 +0000 UTC m=+33.873151645" watchObservedRunningTime="2026-10-30 12:32:16.346114471 +0000 UTC m=+34.797877611"
Oct 30 12:32:16.347156 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.347132    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cqc2z"]
Oct 30 12:32:16.350927 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:32:16.350897    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c865c9_11ec_4356_b33c_b0e4d2e6a401.slice/crio-5c8382466d9efdc516aec2261a625bfecee152b91d6fd121bb1ce9abfa6002c7 WatchSource:0}: Error finding container 5c8382466d9efdc516aec2261a625bfecee152b91d6fd121bb1ce9abfa6002c7: Status 404 returned error can't find the container with id 5c8382466d9efdc516aec2261a625bfecee152b91d6fd121bb1ce9abfa6002c7
Oct 30 12:32:16.379908 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.379886    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6h57s"]
Oct 30 12:32:16.381673 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.381649    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g" event={"ID":"963ec62f-272a-414d-9542-92273189e836","Type":"ContainerStarted","Data":"3a7d7f0c39c1ee8a25f2edd3b680cf688f5120be242cb104f237b2a9235dc2fa"}
Oct 30 12:32:16.382846 ip-10-0-136-246 kubenswrapper[2571]: W1030 12:32:16.382824    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3349844_7e31_4934_8c31_2a208adbf1a5.slice/crio-c43fe4c2c71154602ace42026694a9d91a26c90f63ff554b836708cdd407d2d7 WatchSource:0}: Error finding container c43fe4c2c71154602ace42026694a9d91a26c90f63ff554b836708cdd407d2d7: Status 404 returned error can't find the container with id c43fe4c2c71154602ace42026694a9d91a26c90f63ff554b836708cdd407d2d7
Oct 30 12:32:16.382941 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.382872    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cqc2z" event={"ID":"a3c865c9-11ec-4356-b33c-b0e4d2e6a401","Type":"ContainerStarted","Data":"5c8382466d9efdc516aec2261a625bfecee152b91d6fd121bb1ce9abfa6002c7"}
Oct 30 12:32:16.421399 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.421361    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g" podStartSLOduration=1.349919873 podStartE2EDuration="4.421349887s" totalImagesPullingTime="3.071430014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:16.420637472 +0000 UTC m=+34.872400622" watchObservedRunningTime="2026-10-30 12:32:16.421349887 +0000 UTC m=+34.873113027"
Oct 30 12:32:16.598779 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:16.598722    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cjg4w_7424da73-f41f-433c-9bae-1402110b3e6d/serve-healthcheck-canary/0.log"
Oct 30 12:32:17.386867 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:17.386824    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6h57s" event={"ID":"a3349844-7e31-4934-8c31-2a208adbf1a5","Type":"ContainerStarted","Data":"c43fe4c2c71154602ace42026694a9d91a26c90f63ff554b836708cdd407d2d7"}
Oct 30 12:32:19.702601 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:19.702563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:19.702601 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:19.702607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:19.703031 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:19.702737    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:19.703031 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:19.702749    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:19.703031 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:19.702813    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls podName:7208585d-b1f1-4d3f-9348-2e4abaaa8e26 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:27.702797837 +0000 UTC m=+46.154560954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls") pod "dns-default-fj9wk" (UID: "7208585d-b1f1-4d3f-9348-2e4abaaa8e26") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:19.703031 ip-10-0-136-246 kubenswrapper[2571]: E1030 12:32:19.702830    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:27.702822 +0000 UTC m=+46.154585117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:20.397268 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:20.397167    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cqc2z" event={"ID":"a3c865c9-11ec-4356-b33c-b0e4d2e6a401","Type":"ContainerStarted","Data":"408cb6cc65c7bfff545517cbde50b491027ac52c49378f5ca1fe67ac45f0949f"}
Oct 30 12:32:20.397455 ip-10-0-136-246 kubenswrapper[2571]: I1030 12:32:20.397319    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:21.883607 ip-10-0-136-246 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 12:32:21.903731 ip-10-0-136-246 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 12:32:21.903932 ip-10-0-136-246 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 12:32:21.903988 ip-10-0-136-246 systemd[1]: kubelet.service: Consumed 2.318s CPU time.
Oct 30 12:32:21.914741 ip-10-0-136-246 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:32:21.993045 ip-10-0-136-246 kubenswrapper[4429]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:32:21.993045 ip-10-0-136-246 kubenswrapper[4429]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:32:21.993045 ip-10-0-136-246 kubenswrapper[4429]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:32:21.993045 ip-10-0-136-246 kubenswrapper[4429]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:32:21.995132 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995056    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:21.995132 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995132    4429 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:21.995132 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995136    4429 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995139    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995142    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995145    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995147    4429 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995150    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995152    4429 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995155    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995157    4429 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995160    4429 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995162    4429 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995165    4429 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995168    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995170    4429 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995173    4429 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995176    4429 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995178    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995181    4429 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995184    4429 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:21.995230 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995186    4429 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995189    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995191    4429 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995193    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995196    4429 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995198    4429 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995201    4429 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995203    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995206    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995210    4429 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995215    4429 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995218    4429 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995222    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995225    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995227    4429 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995231    4429 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995234    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995237    4429 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995239    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995242    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:21.995723 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995245    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995248    4429 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995250    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995253    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995255    4429 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995258    4429 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995260    4429 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995263    4429 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995265    4429 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995268    4429 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995271    4429 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995273    4429 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995277    4429 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995295    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995298    4429 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995301    4429 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995305    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995307    4429 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995310    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:21.996226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995312    4429 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995315    4429 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995318    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995320    4429 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995323    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995325    4429 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995328    4429 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995331    4429 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995333    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995336    4429 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995338    4429 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995342    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995344    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995347    4429 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995349    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995352    4429 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995354    4429 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995357    4429 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995359    4429 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995361    4429 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:21.996716 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995365    4429 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995367    4429 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995369    4429 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995372    4429 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995374    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995377    4429 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995380    4429 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995382    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995385    4429 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995388    4429 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995390    4429 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995393    4429 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995395    4429 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995399    4429 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995403    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995405    4429 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995408    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995411    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995413    4429 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:21.997207 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995416    4429 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995419    4429 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995421    4429 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995423    4429 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995426    4429 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995428    4429 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.995431    4429 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996150    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996157    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996159    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996162    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996165    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996167    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996170    4429 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996173    4429 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996175    4429 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996178    4429 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996181    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996188    4429 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996191    4429 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:21.997701 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996194    4429 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996197    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996200    4429 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996202    4429 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996205    4429 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996207    4429 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996210    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996212    4429 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996216    4429 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996219    4429 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996222    4429 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996225    4429 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996229    4429 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996231    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996234    4429 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996237    4429 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996240    4429 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996243    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996246    4429 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996248    4429 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:21.998249 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996251    4429 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996253    4429 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996255    4429 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996258    4429 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996260    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996263    4429 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996265    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996268    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996270    4429 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996272    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996276    4429 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996278    4429 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996295    4429 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996298    4429 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996301    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996303    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996306    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996308    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996311    4429 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:21.998759 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996313    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996316    4429 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996318    4429 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996321    4429 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996323    4429 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996328    4429 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996332    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996336    4429 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996339    4429 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996342    4429 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996344    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996347    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996349    4429 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996352    4429 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996354    4429 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996357    4429 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996360    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996362    4429 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996365    4429 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996368    4429 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:21.999225 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996370    4429 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996373    4429 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996375    4429 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996378    4429 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996380    4429 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996385    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996388    4429 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996391    4429 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996393    4429 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996396    4429 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996398    4429 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996401    4429 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996403    4429 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996406    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996408    4429 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996411    4429 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996413    4429 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996416    4429 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996418    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996421    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:21.999749 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996423    4429 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996426    4429 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996428    4429 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996431    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996433    4429 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996436    4429 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996439    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996441    4429 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996444    4429 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996447    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996450    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996453    4429 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996455    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996457    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996928    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996934    4429 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996937    4429 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996941    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996944    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:22.000236 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996946    4429 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996949    4429 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996951    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996954    4429 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996956    4429 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996959    4429 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996961    4429 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996964    4429 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996967    4429 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996970    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996972    4429 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996974    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996978    4429 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996981    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996984    4429 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996988    4429 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996990    4429 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996993    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996995    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:22.000774 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.996997    4429 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997000    4429 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997002    4429 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997005    4429 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997008    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997010    4429 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997013    4429 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997015    4429 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997018    4429 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997020    4429 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997022    4429 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997026    4429 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997030    4429 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997032    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997035    4429 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997038    4429 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997042    4429 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997045    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997048    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:22.001583 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997050    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997053    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997055    4429 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997058    4429 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997060    4429 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997062    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997065    4429 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997077    4429 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997080    4429 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997082    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997085    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997088    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997090    4429 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997092    4429 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997096    4429 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997098    4429 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997101    4429 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997104    4429 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997106    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:22.002418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997108    4429 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997111    4429 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997114    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997116    4429 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997119    4429 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997122    4429 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997125    4429 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997128    4429 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997130    4429 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997132    4429 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997135    4429 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997137    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997140    4429 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997142    4429 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997145    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997147    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997150    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997152    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997155    4429 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997157    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:22.002922 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997160    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997164    4429 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997166    4429 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997169    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997171    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997174    4429 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997176    4429 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997179    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997182    4429 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997185    4429 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997187    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997190    4429 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997192    4429 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997195    4429 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997197    4429 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997200    4429 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997202    4429 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997205    4429 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997208    4429 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997211    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:22.003487 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997213    4429 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997216    4429 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997219    4429 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997221    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997341    4429 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997359    4429 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997364    4429 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997369    4429 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997373    4429 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997376    4429 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997381    4429 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997385    4429 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997390    4429 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997393    4429 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997396    4429 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997400    4429 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997403    4429 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997406    4429 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997409    4429 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997412    4429 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997414    4429 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997417    4429 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997422    4429 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:32:22.003988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997424    4429 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997427    4429 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997430    4429 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997434    4429 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997437    4429 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997440    4429 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997444    4429 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997448    4429 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997450    4429 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997453    4429 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997456    4429 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997459    4429 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997462    4429 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997466    4429 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997469    4429 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997472    4429 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997475    4429 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997478    4429 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997481    4429 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997485    4429 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997488    4429 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997493    4429 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997496    4429 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997499    4429 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:32:22.004576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997504    4429 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997507    4429 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997510    4429 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997513    4429 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997516    4429 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997519    4429 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997522    4429 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997524    4429 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997528    4429 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997530    4429 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997533    4429 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997537    4429 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997540    4429 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997543    4429 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997548    4429 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997551    4429 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997554    4429 flags.go:64] FLAG: --help="false"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997557    4429 flags.go:64] FLAG: --hostname-override="ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997560    4429 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997563    4429 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997566    4429 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997569    4429 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997572    4429 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997575    4429 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:32:22.005274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997578    4429 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997581    4429 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997584    4429 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997586    4429 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997589    4429 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997593    4429 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997596    4429 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997599    4429 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997602    4429 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997605    4429 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997608    4429 flags.go:64] FLAG: --lock-file=""
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997610    4429 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997620    4429 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997625    4429 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997630    4429 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997633    4429 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997636    4429 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997639    4429 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997642    4429 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997646    4429 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997649    4429 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997651    4429 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997657    4429 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997660    4429 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997664    4429 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:32:22.005918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997666    4429 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997669    4429 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997672    4429 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997675    4429 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997678    4429 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997681    4429 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997683    4429 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997690    4429 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997693    4429 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997696    4429 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997698    4429 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997702    4429 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997706    4429 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997709    4429 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997712    4429 flags.go:64] FLAG: --port="10250"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997715    4429 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997718    4429 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0423db029ce944085"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997722    4429 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997725    4429 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997728    4429 flags.go:64] FLAG: --register-node="true"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997731    4429 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997734    4429 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997737    4429 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997740    4429 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997742    4429 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997746    4429 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:32:22.006550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997749    4429 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997752    4429 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997755    4429 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997759    4429 flags.go:64] FLAG: --runonce="false"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997761    4429 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997764    4429 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997768    4429 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997771    4429 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997774    4429 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997777    4429 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997780    4429 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997783    4429 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997785    4429 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997788    4429 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997791    4429 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997794    4429 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997796    4429 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997799    4429 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997804    4429 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997809    4429 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997812    4429 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997815    4429 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997819    4429 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997822    4429 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997825    4429 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:32:22.007211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997828    4429 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997831    4429 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997834    4429 flags.go:64] FLAG: --v="2"
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997838    4429 flags.go:64] FLAG: --version="false"
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997842    4429 flags.go:64] FLAG: --vmodule=""
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997846    4429 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.997849    4429 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.997997    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998001    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998003    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998007    4429 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998010    4429 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998013    4429 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998016    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998019    4429 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998021    4429 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998024    4429 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998026    4429 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998029    4429 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998031    4429 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998034    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:22.007919 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998036    4429 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998039    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998041    4429 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998044    4429 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998048    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998050    4429 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998053    4429 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998055    4429 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998060    4429 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998064    4429 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998067    4429 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998073    4429 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998076    4429 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998079    4429 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998081    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998084    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998087    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998089    4429 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998092    4429 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:22.008542 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998094    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998097    4429 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998099    4429 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998103    4429 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998107    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998109    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998112    4429 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998115    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998118    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998121    4429 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998123    4429 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998126    4429 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998128    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998131    4429 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998134    4429 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998136    4429 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998139    4429 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998143    4429 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998145    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998148    4429 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:22.009137 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998151    4429 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998153    4429 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998156    4429 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998158    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998161    4429 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998169    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998173    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998175    4429 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998178    4429 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998180    4429 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998183    4429 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998185    4429 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998188    4429 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998190    4429 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998193    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998195    4429 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998197    4429 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998200    4429 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998203    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998205    4429 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:22.009752 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998208    4429 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998210    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998212    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998215    4429 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998217    4429 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998220    4429 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998222    4429 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998225    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998228    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998231    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998234    4429 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998236    4429 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998239    4429 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998243    4429 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998245    4429 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998248    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998250    4429 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998253    4429 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998263    4429 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:22.010337 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998266    4429 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998268    4429 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998271    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998273    4429 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998276    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998278    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998295    4429 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998298    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998301    4429 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998303    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998306    4429 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998308    4429 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998311    4429 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998314    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:21.998342    4429 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998385    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:22.010954 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998390    4429 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998393    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998396    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998399    4429 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998401    4429 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998404    4429 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998407    4429 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998410    4429 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998413    4429 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998415    4429 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998419    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998422    4429 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998424    4429 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998427    4429 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998429    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998432    4429 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998440    4429 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998442    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998445    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998448    4429 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:22.011441 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998450    4429 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998452    4429 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998455    4429 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998458    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998460    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998463    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998465    4429 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998468    4429 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998470    4429 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998472    4429 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998475    4429 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998477    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998480    4429 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998482    4429 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998485    4429 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998487    4429 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998489    4429 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998492    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998494    4429 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:22.012156 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998498    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998500    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998503    4429 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998507    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998509    4429 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998512    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998515    4429 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998517    4429 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998520    4429 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998522    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998530    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998533    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998536    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998539    4429 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998541    4429 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998544    4429 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998546    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998549    4429 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998551    4429 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:22.013102 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998554    4429 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998556    4429 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998559    4429 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998561    4429 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998563    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998566    4429 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998568    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998571    4429 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998573    4429 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998575    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998578    4429 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998581    4429 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998583    4429 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998595    4429 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998598    4429 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998602    4429 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998605    4429 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998608    4429 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998610    4429 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:22.013987 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998613    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998615    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998618    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998621    4429 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998623    4429 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998626    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998629    4429 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998631    4429 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998635    4429 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998639    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998641    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998644    4429 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998647    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998649    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998652    4429 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998654    4429 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998657    4429 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998659    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998662    4429 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:22.014805 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998664    4429 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998667    4429 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998669    4429 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998672    4429 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998674    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998676    4429 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998680    4429 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998684    4429 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:21.998687    4429 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.003675    4429 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.003692    4429 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.003711    4429 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.003716    4429 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003775    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003779    4429 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003783    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003787    4429 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003790    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003793    4429 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003795    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:22.015651 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003798    4429 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003800    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003803    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003805    4429 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003808    4429 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003810    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003813    4429 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003816    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003818    4429 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003821    4429 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003823    4429 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003826    4429 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003828    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003831    4429 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003834    4429 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003836    4429 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003838    4429 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003841    4429 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003844    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003846    4429 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:22.016714 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003849    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003852    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003856    4429 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003860    4429 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003863    4429 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003866    4429 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003869    4429 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003871    4429 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003874    4429 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003876    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003879    4429 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003882    4429 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003884    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003887    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003889    4429 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003892    4429 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003894    4429 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003897    4429 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003899    4429 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:22.017663 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003901    4429 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003904    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003907    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003910    4429 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003912    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003915    4429 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003917    4429 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003920    4429 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003924    4429 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003928    4429 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003932    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003935    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003937    4429 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003940    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003942    4429 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003945    4429 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003948    4429 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003951    4429 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003953    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:22.018546 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003955    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003958    4429 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003960    4429 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003963    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003965    4429 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003968    4429 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003971    4429 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003974    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003976    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003979    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003981    4429 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003984    4429 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003986    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003989    4429 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003992    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003995    4429 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.003998    4429 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004001    4429 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004003    4429 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:22.019544 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004006    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004009    4429 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004012    4429 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004014    4429 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004016    4429 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004019    4429 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004021    4429 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004024    4429 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004026    4429 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004029    4429 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004032    4429 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004034    4429 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004037    4429 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004039    4429 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004042    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004044    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004047    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004049    4429 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004052    4429 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004054    4429 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:22.020435 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004056    4429 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004059    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.004086    4429 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004224    4429 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004228    4429 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004231    4429 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004234    4429 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004237    4429 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004240    4429 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004242    4429 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004245    4429 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004247    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004250    4429 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004253    4429 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004255    4429 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004258    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:22.021323 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004260    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004263    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004265    4429 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004268    4429 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004270    4429 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004273    4429 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004275    4429 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004278    4429 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004296    4429 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004299    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004301    4429 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004304    4429 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004307    4429 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004309    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004312    4429 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004314    4429 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004317    4429 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004320    4429 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004322    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004325    4429 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:22.021988 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004327    4429 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004330    4429 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004333    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004335    4429 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004338    4429 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004341    4429 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004344    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004346    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004350    4429 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004353    4429 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004356    4429 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004359    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004361    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004364    4429 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004366    4429 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004369    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004371    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004374    4429 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004376    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:22.022858 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004379    4429 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004381    4429 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004384    4429 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004386    4429 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004389    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004391    4429 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004394    4429 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004396    4429 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004399    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004402    4429 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004404    4429 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004406    4429 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004409    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004411    4429 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004414    4429 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004416    4429 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004419    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004421    4429 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004424    4429 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004427    4429 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:22.023693 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004430    4429 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004432    4429 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004435    4429 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004437    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004440    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004442    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004444    4429 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004447    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004449    4429 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004452    4429 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004454    4429 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004457    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004460    4429 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004462    4429 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004465    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004467    4429 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004470    4429 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004472    4429 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004474    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:22.024578 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004477    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004479    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004482    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004484    4429 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004487    4429 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004489    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004492    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004494    4429 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004497    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004499    4429 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004503    4429 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004507    4429 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004511    4429 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004514    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004516    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:22.025503 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.004540    4429 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004571    4429 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004575    4429 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004578    4429 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004581    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004583    4429 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004586    4429 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004589    4429 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004592    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004594    4429 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004597    4429 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004599    4429 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004602    4429 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004604    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004608    4429 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004612    4429 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004615    4429 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004618    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004621    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004624    4429 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:32:22.026162 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004626    4429 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004629    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004631    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004634    4429 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004636    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004639    4429 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004641    4429 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004644    4429 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004647    4429 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004649    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004652    4429 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004655    4429 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004658    4429 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004660    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004663    4429 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004665    4429 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004667    4429 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004670    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004672    4429 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:32:22.027094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004675    4429 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004680    4429 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004682    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004684    4429 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004687    4429 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004689    4429 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004692    4429 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004694    4429 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004696    4429 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004699    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004701    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004704    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004710    4429 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004712    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004715    4429 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004717    4429 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004720    4429 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004722    4429 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004725    4429 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004728    4429 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:32:22.028033 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004730    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004733    4429 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004735    4429 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004738    4429 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004740    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004744    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004746    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004749    4429 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004751    4429 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004754    4429 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004757    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004759    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004762    4429 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004768    4429 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004771    4429 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004775    4429 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004778    4429 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004782    4429 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004785    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:32:22.028928 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004788    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004791    4429 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004794    4429 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004796    4429 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004799    4429 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004803    4429 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004806    4429 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004808    4429 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004811    4429 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004813    4429 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004816    4429 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004818    4429 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004821    4429 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004824    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004826    4429 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004829    4429 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004831    4429 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004834    4429 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004837    4429 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004839    4429 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:32:22.029859 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004842    4429 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004845    4429 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004847    4429 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004850    4429 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004852    4429 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004855    4429 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004859    4429 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004861    4429 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:22.004864    4429 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.004920    4429 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.005892    4429 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.005942    4429 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.006113    4429 server.go:1014] "Starting client certificate rotation"
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.006254    4429 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.006311    4429 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:26:42 +0000 UTC" deadline="2028-08-12 19:33:56.82244329 +0000 UTC"
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.006352    4429 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15655h1m34.816093521s"
Oct 30 12:32:22.030834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.006582    4429 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:32:22.031586 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.007678    4429 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:32:22.031586 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.009035    4429 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:32:22.031586 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.010543    4429 log.go:25] "Validated CRI v1 image API"
Oct 30 12:32:22.031586 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.011259    4429 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:32:22.031586 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.013232    4429 fs.go:135] Filesystem UUIDs: map[0d3af223-41f0-4441-8f69-dd57254ecba2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fc874164-8534-4914-979b-485db38c5142:/dev/nvme0n1p3]
Oct 30 12:32:22.031839 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.013261    4429 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0593874be2f59400ad0aa1d2854eb6e0d905458c37398f0c1f8c9f41c53221eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0593874be2f59400ad0aa1d2854eb6e0d905458c37398f0c1f8c9f41c53221eb/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12162cd4b5e123cd344dfe97b5c3f483030c25c10058d97848ab57de8a38d0b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12162cd4b5e123cd344dfe97b5c3f483030c25c10058d97848ab57de8a38d0b4/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dcebe8bb23d1172a1ccbb0b8c208c7eaa2248edaf7719ef15e1d52987dab2d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dcebe8bb23d1172a1ccbb0b8c208c7eaa2248edaf7719ef15e1d52987dab2d4/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55f1176799472fe09027b81fa69da97e56aeafd1bfeb5b08ec65d48961c1f2bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55f1176799472fe09027b81fa69da97e56aeafd1bfeb5b08ec65d48961c1f2bd/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c8382466d9efdc516aec2261a625bfecee152b91d6fd121bb1ce9abfa6002c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c8382466d9efdc516aec2261a625bfecee152b91d6fd121bb1ce9abfa6002c7/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e9c85b3c5dcf82d1181b64cc6515f91f2ca019d65ee50dd806cc3cdf49f5c58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e9c85b3c5dcf82d1181b64cc6515f91f2ca019d65ee50dd806cc3cdf49f5c58/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87d78bfd9709d2f7366d2667c96a089f3f7488515a47a35a9b286a37b3811bf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87d78bfd9709d2f7366d2667c96a089f3f7488515a47a35a9b286a37b3811bf4/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99a5403e2faa3a70828bc4291407dcc76d07127f0f7965a89f5baa72ca968060/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99a5403e2faa3a70828bc4291407dcc76d07127f0f7965a89f5baa72ca968060/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adf93a4a12fbee1e7d7304122d390fe16a8007a6a1cb09c9d8b0a6d23b0d04a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adf93a4a12fbee1e7d7304122d390fe16a8007a6a1cb09c9d8b0a6d23b0d04a2/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0766ac3619e522c5a0703fd965d57fc9d07d4e8a6e6e306063eb659bae81e02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0766ac3619e522c5a0703fd965d57fc9d07d4e8a6e6e306063eb659bae81e02/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbf1d4cb6e81c96ffcc9c384d22760f6393a5d852cb1979b45aac97bf2137320/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbf1d4cb6e81c96ffcc9c384d22760f6393a5d852cb1979b45aac97bf2137320/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c43fe4c2c71154602ace42026694a9d91a26c90f63ff554b836708cdd407d2d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c43fe4c2c71154602ace42026694a9d91a26c90f63ff554b836708cdd407d2d7/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb77d6c826e15d3b10bafd03931dde3e18da3def548763d58e124f21237f010d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb77d6c826e15d3b10bafd03931dde3e18da3def548763d58e124f21237f010d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8715811d525446c4b76e20708251cafeefd1e7a20fd5af1ae5269ac58ec4944/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8715811d525446c4b76e20708251cafeefd1e7a20fd5af1ae5269ac58ec4944/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37a32307-3434-4dff-91d1-8dd3e6c8369e/volumes/kubernetes.io~projected/kube-api-access-nvftl:{mountpoint:/var/lib/kubelet/pods/37a32307-3434-4dff-91d1-8dd3e6c8369e/volumes/kubernetes.io~projected/kube-api-access-nvftl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4825a637-bc1a-4b66-acd1-c0081284ae23/volumes/kubernetes.io~projected/kube-api-access-ztsmb:{mountpoint:/var/lib/kubelet/pods/4825a637-bc1a-4b66-acd1-c0081284ae23/volumes/kubernetes.io~projected/kube-api-access-ztsmb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bc2566d-5249-44ca-be3a-697d3eb4d603/volumes/kubernetes.io~projected/kube-api-access-j75mt:{mountpoint:/var/lib/kubelet/pods/4bc2566d-5249-44ca-be3a-697d3eb4d603/volumes/kubernetes.io~projected/kube-api-access-j75mt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bf513a7-048e-42d8-98fb-5b7adbc09011/volumes/kubernetes.io~projected/kube-api-access-zx8h5:{mountpoint:/var/lib/kubelet/pods/6bf513a7-048e-42d8-98fb-5b7adbc09011/volumes/kubernetes.io~projected/kube-api-access-zx8h5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7208585d-b1f1-4d3f-9348-2e4abaaa8e26/volumes/kubernetes.io~projected/kube-api-access-ksxpk:{mountpoint:/var/lib/kubelet/pods/7208585d-b1f1-4d3f-9348-2e4abaaa8e26/volumes/kubernetes.io~projected/kube-api-access-ksxpk major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7424da73-f41f-433c-9bae-1402110b3e6d/volumes/kubernetes.io~projected/kube-api-access-9qssl:{mountpoint:/var/lib/kubelet/pods/7424da73-f41f-433c-9bae-1402110b3e6d/volumes/kubernetes.io~projected/kube-api-access-9qssl major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7424da73-f41f-433c-9bae-1402110b3e6d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7424da73-f41f-433c-9bae-1402110b3e6d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e07537a-35f5-40ad-b6fe-6b0c92717cf4/volumes/kubernetes.io~projected/kube-api-access-cmjh8:{mountpoint:/var/lib/kubelet/pods/8e07537a-35f5-40ad-b6fe-6b0c92717cf4/volumes/kubernetes.io~projected/kube-api-access-cmjh8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/963ec62f-272a-414d-9542-92273189e836/volumes/kubernetes.io~projected/kube-api-access-wv4xr:{mountpoint:/var/lib/kubelet/pods/963ec62f-272a-414d-9542-92273189e836/volumes/kubernetes.io~projected/kube-api-access-wv4xr major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/963ec62f-272a-414d-9542-92273189e836/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/963ec62f-272a-414d-9542-92273189e836/volumes/kubernetes.io~secret/signing-key major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volumes/kubernetes.io~projected/kube-api-access-6w7cq:{mountpoint:/var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volumes/kubernetes.io~projected/kube-api-access-6w7cq major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3349844-7e31-4934-8c31-2a208adbf1a5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a3349844-7e31-4934-8c31-2a208adbf1a5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3c865c9-11ec-4356-b33c-b0e4d2e6a401/volumes/kubernetes.io~projected/kube-api-access-djhzw:{mountpoint:/var/lib/kubelet/pods/a3c865c9-11ec-4356-b33c-b0e4d2e6a401/volumes/kubernetes.io~projected/kube-api-access-djhzw major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~projected/kube-api-access-9dqfh:{mountpoint:/var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~projected/kube-api-access-9dqfh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cee19c40-c94c-45cd-8deb-16e8a965e053/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cee19c40-c94c-45cd-8deb-16e8a965e053/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1f03eb8-15da-4d44-bc9c-8e043931b8ee/volumes/kubernetes.io~projected/kube-api-access-7w7dm:{mountpoint:/var/lib/kubelet/pods/f1f03eb8-15da-4d44-bc9c-8e043931b8ee/volumes/kubernetes.io~projected/kube-api-access-7w7dm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2102d05-9eb2-4641-a057-7b02ae7d88bd/volumes/kubernetes.io~projected/kube-api-access-rtxsh:{mountpoint:/var/lib/kubelet/pods/f2102d05-9eb2-4641-a057-7b02ae7d88bd/volumes/kubernetes.io~projected/kube-api-access-rtxsh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f35180a3-5aa1-4885-a824-ea3f40d6546a/volumes/kubernetes.io~projected/kube-api-access-nznhb:{mountpoint:/var/lib/kubelet/pods/f35180a3-5aa1-4885-a824-ea3f40d6546a/volumes/kubernetes.io~projected/kube-api-access-nznhb major:0 minor:202 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ab8ab2dba6b59d97c8c9f722282dde41aac8c13d5ff1090717c4b118d1d62786/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f5cd112f46e9783111e4399a6cc943bd7f0f4aae479fee938103cfc09563207b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b5a6c1e6ec4dc9e1ee17a672e2bedc21a3a8b74a69f2c7b909950ce00236a202/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/bace8cff31947c52a1e0e945107411fbc0f0f39da722a9ace342c821bc6dcc76/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/420d80ff1675644b1c364292131c6d0aafb426c75d76d8286c4c547ed858e141/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a42fbbfd4650c8d41f98315bb51243dc2d6b41d88969990cd30ab41d4527ef06/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ab3becc28528552eda4cc8383af46d3bc65ce02ba5bc30b79d426841f38ef13e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c23026111de6ebec808724ed7f774772a89adb0892fc43b2260aee5d4d9fd40b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5c5a1487106856838967e09b2aa959a3e96905782ebe2af698b08c65118cd187/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/1d0a103628a2f9a5aac362dc0a636993eb3353b4315d366f97e114a9ac5b141a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/016705a449a532f6f8dd781bb979874113343b80b3dcddef5f5610fd28cd5101/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/dd0aee355bca5f795357be5dd2cb3e5f4515abb6b9bb4b8e3b90efc6c4e8aec5/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/aa3fb00769c025fa1235b2394e8072f21e5f0fae06b747452598bbd4a61221b2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1aef8472dfa1b993e05803586a97218f2054e21ed6978e2cf08a954e356dd1d6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f84a3ff90d121a26bd7984e81483051336aaedadf6e1af0369e7cf15b6304920/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9524ca74f9509c39ea2f7228aea7f1913613be8fbd24c47e3471a62460f36b6d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d40688e32478949539f538f79b6a5498c0d7bf160f81e14d786766627db82d41/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a21d8446d0f464889ca62d6e2c534afb19edd6a45e336d517647ec52b806c169/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/16d46c91cd1defb74b8de9a7c2b9f4e5fef01f6c8553bfb884bfbfbc734dd8eb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c621813a17ed7747ccf7fa392cab0b0cab5ae831e76d8a370bfed11263d557d3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4d2233abde59c102ef5fa5dbd093128ee53a91eba6328527272a59dfffe5aa2e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/567fdc7e39f094638c742e7ee33aa0276464fccac4756ffdef07e7b5ccfacc19/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/e7f2981c0125fb2d7c5d24b328a01be3584eff80990ab1c6aa68415154252c03/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/b1154c62fae6eb39ff9ee69844806879a86cd0994391d50829ce00c21f183613/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/fb96f7fee04bbc6c1ca22d15c028605dbe1139d3f039872a656efe6f5a871f85/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/6f2e599a6ffd0772c61819974e70d991a17eab2ed35b0725a2db042bea22f958/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/1ea07c0bf93c064628d1718cf9521696b9cc9cd23e4ca0ba60a624a6086f0487/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/8165b5896318bb3863fbbfeccc86f0812256b36872e1675d36efc80cba08b04e/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/8ca5446f21bf44985f31b2aa3525559eeb84a68f7ee0f12aa9ba30d7fc300c85/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5c85aafb6f0386181cbf4649155ad56d50b253785fb1726278e46d50861b9343/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8161d78dd4e31f349398f7553735d9b49afe2141752a4ba76e2179e68d6cc1a5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/414c0e8d6c881872288631a67a1382ccbb776f796ba5251718641fc14631ea0b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8197aed9e40c60ced87b0b605573ea713b18e6163f86c999cc452f78ed84f6ea/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a2853dfc9fb2b5fd468e381c96697d88425b9f67d64d63baa9ef490889fd5d5c/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a4065834d57d8ac978d0fea3ab4a0f3986e3b4c93330217794969de6ad1d047a/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/7ecd97da26236bd112ce1924f595dff893f778186a39fc0ca7968fc0f8e82643/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/31fe8574b2c14ded0526efd5466058426a80a097e52a7c7cbb7a8984ec68ce9c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1c4e43fd32554f2851c3bf44247f49d6017fb10655b773eea8673f7b912157ec/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d753f89013d3c811282b16ec35157b3c86e488542559b89ebcce7ec417976be0/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.022511    4429 manager.go:217] Machine: {Timestamp:2026-10-30 12:32:22.021486469 +0000 UTC m=+0.076980285 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099415 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec248f378efa70f7ff5f699330a040fd SystemUUID:ec248f37-8efa-70f7-ff5f-699330a040fd BootID:72a3a50f-696b-4435-a838-37eb3eb12fa5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0766ac3619e522c5a0703fd965d57fc9d07d4e8a6e6e306063eb659bae81e02/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3c865c9-11ec-4356-b33c-b0e4d2e6a401/volumes/kubernetes.io~projected/kube-api-access-djhzw DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e9c85b3c5dcf82d1181b64cc6515f91f2ca019d65ee50dd806cc3cdf49f5c58/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adf93a4a12fbee1e7d7304122d390fe16a8007a6a1cb09c9d8b0a6d23b0d04a2/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/963ec62f-272a-414d-9542-92273189e836/volumes/kubernetes.io~projected/kube-api-access-wv4xr DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4825a637-bc1a-4b66-acd1-c0081284ae23/volumes/kubernetes.io~projected/kube-api-access-ztsmb DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55f1176799472fe09027b81fa69da97e56aeafd1bfeb5b08ec65d48961c1f2bd/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volumes/kubernetes.io~projected/kube-api-access-6w7cq DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f1f03eb8-15da-4d44-bc9c-8e043931b8ee/volumes/kubernetes.io~projected/kube-api-access-7w7dm DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/37a32307-3434-4dff-91d1-8dd3e6c8369e/volumes/kubernetes.io~projected/kube-api-access-nvftl DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dcebe8bb23d1172a1ccbb0b8c208c7eaa2248edaf7719ef15e1d52987dab2d4/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb77d6c826e15d3b10bafd03931dde3e18da3def548763d58e124f21237f010d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7424da73-f41f-433c-9bae-1402110b3e6d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3349844-7e31-4934-8c31-2a208adbf1a5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8715811d525446c4b76e20708251cafeefd1e7a20fd5af1ae5269ac58ec4944/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9729abbf-37cc-43cb-bbf9-8c08a18eb37a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8e07537a-35f5-40ad-b6fe-6b0c92717cf4/volumes/kubernetes.io~projected/kube-api-access-cmjh8 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99a5403e2faa3a70828bc4291407dcc76d07127f0f7965a89f5baa72ca968060/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bc2566d-5249-44ca-be3a-697d3eb4d603/volumes/kubernetes.io~projected/kube-api-access-j75mt DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0593874be2f59400ad0aa1d2854eb6e0d905458c37398f0c1f8c9f41c53221eb/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12162cd4b5e123cd344dfe97b5c3f483030c25c10058d97848ab57de8a38d0b4/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbf1d4cb6e81c96ffcc9c384d22760f6393a5d852cb1979b45aac97bf2137320/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7208585d-b1f1-4d3f-9348-2e4abaaa8e26/volumes/kubernetes.io~projected/kube-api-access-ksxpk DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/963ec62f-272a-414d-9542-92273189e836/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bf513a7-048e-42d8-98fb-5b7adbc09011/volumes/kubernetes.io~projected/kube-api-access-zx8h5 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cee19c40-c94c-45cd-8deb-16e8a965e053/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f2102d05-9eb2-4641-a057-7b02ae7d88bd/volumes/kubernetes.io~projected/kube-api-access-rtxsh DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87d78bfd9709d2f7366d2667c96a089f3f7488515a47a35a9b286a37b3811bf4/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7424da73-f41f-433c-9bae-1402110b3e6d/volumes/kubernetes.io~projected/kube-api-access-9qssl DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/adf5def0-890c-405c-9706-664675f53547/volumes/kubernetes.io~projected/kube-api-access-9dqfh DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f35180a3-5aa1-4885-a824-ea3f40d6546a/volumes/kubernetes.io~projected/kube-api-access-nznhb DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c43fe4c2c71154602ace42026694a9d91a26c90f63ff554b836708cdd407d2d7/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c8382466d9efdc516aec2261a625bfecee152b91d6fd121bb1ce9abfa6002c7/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:12162cd4b5e123c MacAddress:32:dd:50:1a:09:4b Speed:10000 Mtu:8901} {Name:2dcebe8bb23d117 MacAddress:7e:49:88:a7:2c:36 Speed:10000 Mtu:8901} {Name:5c8382466d9efdc MacAddress:32:d8:c1:d5:5e:d0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:0f:c8:75 Speed:0 Mtu:9001} {Name:br-int MacAddress:9e:1c:13:bf:5c:a9 Speed:0 Mtu:8901} {Name:c43fe4c2c711546 MacAddress:5e:0d:46:de:f7:c0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:0f:c8:75 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7e:d3:d5:9a:40:74 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:6c:0e:f3:86:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.022940    4429 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.022998    4429 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.023325    4429 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.023350    4429 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-246.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.023530    4429 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.023542    4429 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.023561    4429 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.023589    4429 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.023687    4429 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.023954    4429 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.024104    4429 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.024152    4429 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.024173    4429 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.024187    4429 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.024202    4429 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.024215    4429 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.024997    4429 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.025010    4429 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.025570    4429 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.028401    4429 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.028415    4429 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.028536    4429 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.028801    4429 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.028847    4429 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.028965    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.028984    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029004    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029014    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029024    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029034    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029045    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029055    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029067    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029077    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029096    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029110    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029149    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:32:22.032085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.029159    4429 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:32:22.033868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.033556    4429 server.go:1263] "Started kubelet"
Oct 30 12:32:22.033868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.033636    4429 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:32:22.033868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.033651    4429 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:32:22.033868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.033757    4429 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:32:22.034097 ip-10-0-136-246 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:32:22.034203 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.034167    4429 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:32:22.035205 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.035169    4429 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:32:22.037525 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.036758    4429 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:22.038354    4429 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-246.ec2.internal\" not found"
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.038421    4429 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.038432    4429 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.038802    4429 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.038857    4429 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.038848    4429 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:26:44 +0000 UTC" deadline="2028-07-23 16:10:18.5995233 +0000 UTC"
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.038883    4429 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15171h37m56.560643652s"
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040457    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963ec62f-272a-414d-9542-92273189e836" volumeName="kubernetes.io/configmap/963ec62f-272a-414d-9542-92273189e836-signing-cabundle" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040500    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963ec62f-272a-414d-9542-92273189e836" volumeName="kubernetes.io/secret/963ec62f-272a-414d-9542-92273189e836-signing-key" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040515    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c865c9-11ec-4356-b33c-b0e4d2e6a401" volumeName="kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040528    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee19c40-c94c-45cd-8deb-16e8a965e053" volumeName="kubernetes.io/configmap/cee19c40-c94c-45cd-8deb-16e8a965e053-konnectivity-ca" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040548    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a32307-3434-4dff-91d1-8dd3e6c8369e" volumeName="kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040562    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a32307-3434-4dff-91d1-8dd3e6c8369e" volumeName="kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040579    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4825a637-bc1a-4b66-acd1-c0081284ae23" volumeName="kubernetes.io/projected/4825a637-bc1a-4b66-acd1-c0081284ae23-kube-api-access-ztsmb" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040593    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bf513a7-048e-42d8-98fb-5b7adbc09011" volumeName="kubernetes.io/configmap/6bf513a7-048e-42d8-98fb-5b7adbc09011-serviceca" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040615    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7424da73-f41f-433c-9bae-1402110b3e6d" volumeName="kubernetes.io/secret/7424da73-f41f-433c-9bae-1402110b3e6d-cert" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040627    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee19c40-c94c-45cd-8deb-16e8a965e053" volumeName="kubernetes.io/secret/cee19c40-c94c-45cd-8deb-16e8a965e053-agent-certs" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040641    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a32307-3434-4dff-91d1-8dd3e6c8369e" volumeName="kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040659    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a32307-3434-4dff-91d1-8dd3e6c8369e" volumeName="kubernetes.io/projected/37a32307-3434-4dff-91d1-8dd3e6c8369e-kube-api-access-nvftl" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040672    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7208585d-b1f1-4d3f-9348-2e4abaaa8e26" volumeName="kubernetes.io/empty-dir/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-tmp-dir" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040697    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e07537a-35f5-40ad-b6fe-6b0c92717cf4" volumeName="kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040715    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9729abbf-37cc-43cb-bbf9-8c08a18eb37a" volumeName="kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-env-overrides" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040732    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9729abbf-37cc-43cb-bbf9-8c08a18eb37a" volumeName="kubernetes.io/secret/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040744    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7208585d-b1f1-4d3f-9348-2e4abaaa8e26" volumeName="kubernetes.io/projected/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-kube-api-access-ksxpk" seLinuxMountContext=""
Oct 30 12:32:22.040763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040774    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9729abbf-37cc-43cb-bbf9-8c08a18eb37a" volumeName="kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-config" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040802    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3349844-7e31-4934-8c31-2a208adbf1a5" volumeName="kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040816    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf5def0-890c-405c-9706-664675f53547" volumeName="kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-tmp" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040828    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1f03eb8-15da-4d44-bc9c-8e043931b8ee" volumeName="kubernetes.io/projected/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kube-api-access-7w7dm" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040844    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35180a3-5aa1-4885-a824-ea3f40d6546a" volumeName="kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040857    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35180a3-5aa1-4885-a824-ea3f40d6546a" volumeName="kubernetes.io/empty-dir/f35180a3-5aa1-4885-a824-ea3f40d6546a-data-volume" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040873    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bc2566d-5249-44ca-be3a-697d3eb4d603" volumeName="kubernetes.io/configmap/4bc2566d-5249-44ca-be3a-697d3eb4d603-iptables-alerter-script" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040885    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf5def0-890c-405c-9706-664675f53547" volumeName="kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-etc-tuned" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040898    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35180a3-5aa1-4885-a824-ea3f40d6546a" volumeName="kubernetes.io/projected/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-api-access-nznhb" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040915    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4825a637-bc1a-4b66-acd1-c0081284ae23" volumeName="kubernetes.io/empty-dir/4825a637-bc1a-4b66-acd1-c0081284ae23-tmp-dir" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040927    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7208585d-b1f1-4d3f-9348-2e4abaaa8e26" volumeName="kubernetes.io/configmap/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-config-volume" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040955    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2102d05-9eb2-4641-a057-7b02ae7d88bd" volumeName="kubernetes.io/projected/f2102d05-9eb2-4641-a057-7b02ae7d88bd-kube-api-access-rtxsh" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040973    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bc2566d-5249-44ca-be3a-697d3eb4d603" volumeName="kubernetes.io/projected/4bc2566d-5249-44ca-be3a-697d3eb4d603-kube-api-access-j75mt" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.040987    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963ec62f-272a-414d-9542-92273189e836" volumeName="kubernetes.io/projected/963ec62f-272a-414d-9542-92273189e836-kube-api-access-wv4xr" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041001    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9729abbf-37cc-43cb-bbf9-8c08a18eb37a" volumeName="kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041019    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9729abbf-37cc-43cb-bbf9-8c08a18eb37a" volumeName="kubernetes.io/projected/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-kube-api-access-6w7cq" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041034    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf5def0-890c-405c-9706-664675f53547" volumeName="kubernetes.io/projected/adf5def0-890c-405c-9706-664675f53547-kube-api-access-9dqfh" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041051    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bf513a7-048e-42d8-98fb-5b7adbc09011" volumeName="kubernetes.io/projected/6bf513a7-048e-42d8-98fb-5b7adbc09011-kube-api-access-zx8h5" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041064    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7424da73-f41f-433c-9bae-1402110b3e6d" volumeName="kubernetes.io/projected/7424da73-f41f-433c-9bae-1402110b3e6d-kube-api-access-9qssl" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041077    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e07537a-35f5-40ad-b6fe-6b0c92717cf4" volumeName="kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-daemon-config" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041095    4429 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e07537a-35f5-40ad-b6fe-6b0c92717cf4" volumeName="kubernetes.io/projected/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-kube-api-access-cmjh8" seLinuxMountContext=""
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041112    4429 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:32:22.041957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.041120    4429 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:32:22.042793 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.042316    4429 factory.go:55] Registering systemd factory
Oct 30 12:32:22.042793 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.042333    4429 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:32:22.044689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.043088    4429 factory.go:153] Registering CRI-O factory
Oct 30 12:32:22.044689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.043108    4429 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:32:22.044689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.043161    4429 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:32:22.044689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.043181    4429 factory.go:103] Registering Raw factory
Oct 30 12:32:22.044689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.043198    4429 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:32:22.044689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.043535    4429 manager.go:319] Starting recovery of all containers
Oct 30 12:32:22.065539 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.065509    4429 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:32:22.066877 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.066860    4429 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:32:22.066877 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.066879    4429 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:32:22.067044 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.066899    4429 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:32:22.067044 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:22.066939    4429 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 12:32:22.071735 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.071704    4429 generic.go:356] "Generic (PLEG): container finished" podID="3a7fe78ff9a981c06dfba04265b1aa94" containerID="8e31d351c9871cbb7780a40cd7f0e934d63e1c562de7ad87eb34348b37538b85" exitCode=0
Oct 30 12:32:22.091252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.091227    4429 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 12:32:22.092038 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.092019    4429 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:32:22.098299 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.098256    4429 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="ac3879fe50009fbf8d3cd395345219667d8b1f4911d62846062dc8aa86a57695" exitCode=0
Oct 30 12:32:22.098391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.098315    4429 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="466eb53c5ff85ecd5cbc3b64732bcbcc50575d629db08b89b6a2ce67d28614ec" exitCode=0
Oct 30 12:32:22.098391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.098326    4429 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="bb2ddbba56eff50eadb473ec6d021bcf015bad16ce4b8643d6d5b10aef4163d8" exitCode=0
Oct 30 12:32:22.098391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.098341    4429 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="4a925a5a1e12cfda231bacbe423ab29f245c7dc9235bbe65d9bb98502c2c5ff1" exitCode=0
Oct 30 12:32:22.098391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.098349    4429 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="c067dd615d7ba695901719e05341966aefbe96a28afaf000da4ab3b4418d3d85" exitCode=0
Oct 30 12:32:22.098391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.098357    4429 generic.go:356] "Generic (PLEG): container finished" podID="37a32307-3434-4dff-91d1-8dd3e6c8369e" containerID="d505ee458da66ce5037247f6b83b0eef2698d0677100638e79a18e5ce9107faa" exitCode=0
Oct 30 12:32:22.107001 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.106978    4429 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:32:22.107001 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.106992    4429 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:32:22.107316 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.107266    4429 manager.go:324] Recovery completed
Oct 30 12:32:22.118872 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.118853    4429 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:32:22.118984 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.118968    4429 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:32:22.119918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.119875    4429 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:32:22.120006 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.119927    4429 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:32:22.120006 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.119937    4429 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:32:22.121348 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121332    4429 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:32:22.121348 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121347    4429 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:32:22.121418 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121365    4429 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:32:22.121516 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121500    4429 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 12:32:22.121546 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121519    4429 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 12:32:22.121546 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121541    4429 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 12:32:22.121607 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121549    4429 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 12:32:22.121607 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121554    4429 policy_none.go:50] "Start"
Oct 30 12:32:22.121607 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121563    4429 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:32:22.121607 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.121573    4429 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:32:22.127405 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.127392    4429 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 12:32:22.127443 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.127407    4429 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 12:32:22.127443 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.127413    4429 policy_none.go:44] "Start"
Oct 30 12:32:22.132891 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.132856    4429 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:32:22.137768 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.137730    4429 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:32:22.138279 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:22.137753    4429 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:32:22.138339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.138299    4429 server.go:87] "Starting device plugin registration server"
Oct 30 12:32:22.138534 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.138522    4429 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:32:22.138581 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.138537    4429 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:32:22.138646 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.138629    4429 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:32:22.138794 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.138744    4429 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:32:22.138896 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.138883    4429 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:32:22.138896 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.138895    4429 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:32:22.139030 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.138948    4429 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:32:22.138870269Z","UUID":"c245312a-f821-4d64-aee2-1f77676f5762","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:32:22.139884 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:22.139863    4429 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:32:22.140196 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.140179    4429 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:32:22.140308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.140206    4429 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:32:22.167426 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.167390    4429 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"]
Oct 30 12:32:22.167824 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.167797    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal" event={"ID":"3a7fe78ff9a981c06dfba04265b1aa94","Type":"ContainerStarted","Data":"f2553d278b8a0cff7da05243849c1372e86081eb26c7551720ea53c581841488"}
Oct 30 12:32:22.167896 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.167830    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal" event={"ID":"3a7fe78ff9a981c06dfba04265b1aa94","Type":"ContainerDied","Data":"8e31d351c9871cbb7780a40cd7f0e934d63e1c562de7ad87eb34348b37538b85"}
Oct 30 12:32:22.167896 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.167845    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal" event={"ID":"3a7fe78ff9a981c06dfba04265b1aa94","Type":"ContainerStarted","Data":"eb77d6c826e15d3b10bafd03931dde3e18da3def548763d58e124f21237f010d"}
Oct 30 12:32:22.167896 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.167891    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal" event={"ID":"e9b5f11662f569a9096e79917d2f933f","Type":"ContainerStarted","Data":"3d21452362ce951a37e27b401b77eb14bb87a1ca09b0c426dffeec7ed19195cf"}
Oct 30 12:32:22.168052 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.167903    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal" event={"ID":"e9b5f11662f569a9096e79917d2f933f","Type":"ContainerStarted","Data":"55f1176799472fe09027b81fa69da97e56aeafd1bfeb5b08ec65d48961c1f2bd"}
Oct 30 12:32:22.168052 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.167924    4429 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.168052 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.168025    4429 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.198300 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.198258    4429 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:32:22.198390 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:22.198324    4429 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-246.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.198523 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.198509    4429 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:32:22.198568 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:22.198555    4429 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.238779 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.238755    4429 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:32:22.239759 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.239737    4429 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:32:22.239830 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.239767    4429 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:32:22.239830 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.239778    4429 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-246.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:32:22.239830 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.239809    4429 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.293126 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.293098    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.293126 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.293125    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9b5f11662f569a9096e79917d2f933f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-246.ec2.internal\" (UID: \"e9b5f11662f569a9096e79917d2f933f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.293257 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.293151    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.301383 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.301365    4429 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.301441 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.301417    4429 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.393556 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.393486    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.393556 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.393516    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9b5f11662f569a9096e79917d2f933f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-246.ec2.internal\" (UID: \"e9b5f11662f569a9096e79917d2f933f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.393556 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.393533    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.393746 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.393568    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.393746 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.393604    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e9b5f11662f569a9096e79917d2f933f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-246.ec2.internal\" (UID: \"e9b5f11662f569a9096e79917d2f933f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:22.393746 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:22.393605    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a7fe78ff9a981c06dfba04265b1aa94-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" (UID: \"3a7fe78ff9a981c06dfba04265b1aa94\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:23.024797 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.024760    4429 apiserver.go:52] "Watching apiserver"
Oct 30 12:32:23.070253 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.070221    4429 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:32:23.070717 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.070699    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xz8gl","openshift-cluster-node-tuning-operator/tuned-lgqrc","openshift-dns/node-resolver-fttst","openshift-multus/network-metrics-daemon-kfgnl","kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal","openshift-multus/multus-k4c5j","openshift-dns/dns-default-fj9wk","openshift-ovn-kubernetes/ovnkube-node-2s9ms","openshift-insights/insights-runtime-extractor-gnl9r","openshift-network-operator/iptables-alerter-qmdgt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn","kube-system/global-pull-secret-syncer-6h57s","openshift-multus/multus-additional-cni-plugins-fpqzd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal","openshift-ingress-canary/ingress-canary-cjg4w","openshift-network-diagnostics/network-check-target-cqc2z","openshift-service-ca/service-ca-5fffc97bdf-s7t7g","openshift-image-registry/node-ca-r8lf8"]
Oct 30 12:32:23.072117 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.072097    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:23.075985 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.075967    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.076514 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.076499    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.090510 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090444    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:32:23.090606 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090588    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:32:23.090691 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090668    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.090778 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090724    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.090778 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090757    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.090778 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090767    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.090778 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090777    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:32:23.090955 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090711    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.090955 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.090855    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.091167 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.091142    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:32:23.091167 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.091151    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:32:23.095336 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.095305    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:32:23.095336 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.095326    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v9g2h\""
Oct 30 12:32:23.095458 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.095334    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k87jk\""
Oct 30 12:32:23.095458 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.095346    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5fvrw\""
Oct 30 12:32:23.095458 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.095330    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zprfc\""
Oct 30 12:32:23.095458 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.095409    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fkzfg\""
Oct 30 12:32:23.095624 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.095469    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56d7b\""
Oct 30 12:32:23.095624 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.095478    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:32:23.103571 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.103554    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:32:23.103661 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.103582    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.103661 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.103583    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 12:32:23.103755 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.103668    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kq49t\""
Oct 30 12:32:23.103804 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.103790    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgbt6\""
Oct 30 12:32:23.103848 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.103833    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7nlfh\""
Oct 30 12:32:23.104073 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104053    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:32:23.104119 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104075    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:32:23.104119 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104100    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.104219 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104130    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.104350 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104323    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:32:23.104436 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104368    4429 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:23.104492 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104440    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:32:23.104492 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104445    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:32:23.104608 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.104508    4429 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:23.105122 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.105103    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:32:23.105479 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.105466    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.105767 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.105746    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-69dkl\""
Oct 30 12:32:23.105820 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.105789    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.105861 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.105832    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.105919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.105893    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxtxl\""
Oct 30 12:32:23.106039 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.105987    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:32:23.106165 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.106138    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wqghl\""
Oct 30 12:32:23.106222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.106211    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rclv\""
Oct 30 12:32:23.106406 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.106392    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sr4bw\""
Oct 30 12:32:23.106454 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.106443    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5lgc6\""
Oct 30 12:32:23.108551 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108540    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:32:23.108591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108576    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bklll\""
Oct 30 12:32:23.108792 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108779    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:32:23.108851 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108810    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.108851 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108840    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108848    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108857    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108862    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108887    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108901    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108864    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108932    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108935    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108952    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:32:23.108986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.108983    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:32:23.109302 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.109022    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 12:32:23.109302 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.109055    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:32:23.136481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.136462    4429 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:32:23.136565 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.136523    4429 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-246.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:23.140297 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.140275    4429 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:32:23.142462 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.142441    4429 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:32:23.142531 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.142475    4429 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-246.ec2.internal"
Oct 30 12:32:23.196580 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196555    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-conf\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.196580 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196581    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4825a637-bc1a-4b66-acd1-c0081284ae23-hosts-file\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:32:23.196728 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196603    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-bin\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.196728 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196627    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7cq\" (UniqueName: \"kubernetes.io/projected/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-kube-api-access-6w7cq\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.196728 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196687    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvftl\" (UniqueName: \"kubernetes.io/projected/37a32307-3434-4dff-91d1-8dd3e6c8369e-kube-api-access-nvftl\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.196828 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196747    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-bin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.196858 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196826    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf513a7-048e-42d8-98fb-5b7adbc09011-host\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:32:23.196893 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196878    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.196930 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196909    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysconfig\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.196968 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196948    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-device-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.196997 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.196974    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-tmp-dir\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.197035 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197000    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.197068 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197035    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.197068 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197043    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-tmp-dir\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.197068 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197061    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjh8\" (UniqueName: \"kubernetes.io/projected/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-kube-api-access-cmjh8\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.197157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197083    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8h5\" (UniqueName: \"kubernetes.io/projected/6bf513a7-048e-42d8-98fb-5b7adbc09011-kube-api-access-zx8h5\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:32:23.197157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197106    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75mt\" (UniqueName: \"kubernetes.io/projected/4bc2566d-5249-44ca-be3a-697d3eb4d603-kube-api-access-j75mt\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:32:23.197275 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197257    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-lib-modules\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.197385 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197340    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-systemd-units\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.197437 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197387    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-node-log\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.197437 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197418    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovn-node-metrics-cert\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.197521 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197468    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-netns\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.197601 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197578    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-hostroot\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.197664 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197634    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.197761 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197741    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-netd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.197821 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197780    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-config\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.197821 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197806    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.197821 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197812    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-system-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.197947 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197841    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-k8s-cni-cncf-io\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.197947 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197854    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovn-node-metrics-cert\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.197947 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197897    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-kubelet\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.197947 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197919    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-kubelet-config\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:23.197947 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197945    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-registration-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.198139 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.197983    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-etc-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.198139 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198015    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-tmp\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.198139 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198038    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-config\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.198139 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198076    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.198139 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198105    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxpk\" (UniqueName: \"kubernetes.io/projected/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-kube-api-access-ksxpk\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.198139 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198130    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-script-lib\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.198139 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198133    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-tmp\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.198466 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198161    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-dbus\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:23.198466 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198312    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cee19c40-c94c-45cd-8deb-16e8a965e053-konnectivity-ca\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:32:23.198466 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198353    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f35180a3-5aa1-4885-a824-ea3f40d6546a-crio-socket\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.198466 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198380    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.198466 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198383    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-ovnkube-script-lib\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.198466 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198404    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-multus-certs\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.198466 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198436    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-host\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.198466 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198466    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqfh\" (UniqueName: \"kubernetes.io/projected/adf5def0-890c-405c-9706-664675f53547-kube-api-access-9dqfh\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.198720 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198510    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/963ec62f-272a-414d-9542-92273189e836-signing-key\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:23.198720 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198586    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cni-binary-copy\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.198720 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198672    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cee19c40-c94c-45cd-8deb-16e8a965e053-konnectivity-ca\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:32:23.198720 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198696    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-multus\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.198863 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198725    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-etc-kubernetes\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.198863 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198793    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bf513a7-048e-42d8-98fb-5b7adbc09011-serviceca\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:32:23.198863 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198788    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/963ec62f-272a-414d-9542-92273189e836-signing-key\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:23.198863 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198828    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-slash\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.198863 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198857    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/963ec62f-272a-414d-9542-92273189e836-signing-cabundle\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198879    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cee19c40-c94c-45cd-8deb-16e8a965e053-agent-certs\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198915    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-daemon-config\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198921    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cni-binary-copy\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.198934    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-kubernetes\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199009    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-systemd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199041    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199078    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-daemon-config\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199084    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/963ec62f-272a-414d-9542-92273189e836-signing-cabundle\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:23.199124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199087    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4xr\" (UniqueName: \"kubernetes.io/projected/963ec62f-272a-414d-9542-92273189e836-kube-api-access-wv4xr\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199162    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cee19c40-c94c-45cd-8deb-16e8a965e053-agent-certs\") pod \"konnectivity-agent-xz8gl\" (UID: \"cee19c40-c94c-45cd-8deb-16e8a965e053\") " pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199190    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-cnibin\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199231    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199261    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199321    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7424da73-f41f-433c-9bae-1402110b3e6d-cert\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199314    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3349844-7e31-4934-8c31-2a208adbf1a5-original-pull-secret\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199346    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-var-lib-kubelet\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199398    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7dm\" (UniqueName: \"kubernetes.io/projected/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kube-api-access-7w7dm\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199430    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.199526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199508    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bf513a7-048e-42d8-98fb-5b7adbc09011-serviceca\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199565    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199584    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199596    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-socket-dir-parent\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199601    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7424da73-f41f-433c-9bae-1402110b3e6d-cert\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199632    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f35180a3-5aa1-4885-a824-ea3f40d6546a-data-volume\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199668    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199724    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f35180a3-5aa1-4885-a824-ea3f40d6546a-data-volume\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199728    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:23.199905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199766    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.200186 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.199762    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-system-cni-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.200217 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200201    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-conf-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.200248 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200218    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-socket-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.200248 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200235    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztsmb\" (UniqueName: \"kubernetes.io/projected/4825a637-bc1a-4b66-acd1-c0081284ae23-kube-api-access-ztsmb\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:32:23.200340 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200252    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxsh\" (UniqueName: \"kubernetes.io/projected/f2102d05-9eb2-4641-a057-7b02ae7d88bd-kube-api-access-rtxsh\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:23.200340 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200266    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-kubelet\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.200340 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200322    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-log-socket\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.200471 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200396    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.200471 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200444    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-env-overrides\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.200543 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200482    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-run\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.200543 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200515    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-config-volume\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.200661 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200556    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-var-lib-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.200661 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200599    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-env-overrides\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.200661 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200603    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.200661 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200655    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc2566d-5249-44ca-be3a-697d3eb4d603-host-slash\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:32:23.200852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200668    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-config-volume\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.200852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200677    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qssl\" (UniqueName: \"kubernetes.io/projected/7424da73-f41f-433c-9bae-1402110b3e6d-kube-api-access-9qssl\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:23.200852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200697    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-modprobe-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.200852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200719    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-systemd\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.200852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200784    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-netns\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.200852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200803    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a32307-3434-4dff-91d1-8dd3e6c8369e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.200852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200827    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.201098 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200868    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bc2566d-5249-44ca-be3a-697d3eb4d603-iptables-alerter-script\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:32:23.201098 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200905    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:23.201098 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.200993    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nznhb\" (UniqueName: \"kubernetes.io/projected/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-api-access-nznhb\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.201098 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201041    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-sys\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.201098 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201068    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-etc-tuned\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.201323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201108    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-sys-fs\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.201323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201122    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825a637-bc1a-4b66-acd1-c0081284ae23-tmp-dir\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:32:23.201323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201137    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-ovn\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.201323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201143    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adf5def0-890c-405c-9706-664675f53547-etc-tuned\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.201323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201085    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bc2566d-5249-44ca-be3a-697d3eb4d603-iptables-alerter-script\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:32:23.201323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201175    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cnibin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.201323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.201213    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4825a637-bc1a-4b66-acd1-c0081284ae23-tmp-dir\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:32:23.221257 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.221232    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8h5\" (UniqueName: \"kubernetes.io/projected/6bf513a7-048e-42d8-98fb-5b7adbc09011-kube-api-access-zx8h5\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:32:23.221257 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.221247    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjh8\" (UniqueName: \"kubernetes.io/projected/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-kube-api-access-cmjh8\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.221451 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.221433    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75mt\" (UniqueName: \"kubernetes.io/projected/4bc2566d-5249-44ca-be3a-697d3eb4d603-kube-api-access-j75mt\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:32:23.221500 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.221480    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7cq\" (UniqueName: \"kubernetes.io/projected/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-kube-api-access-6w7cq\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.223659 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.223637    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvftl\" (UniqueName: \"kubernetes.io/projected/37a32307-3434-4dff-91d1-8dd3e6c8369e-kube-api-access-nvftl\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.228252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.228228    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqfh\" (UniqueName: \"kubernetes.io/projected/adf5def0-890c-405c-9706-664675f53547-kube-api-access-9dqfh\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.228436 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.228421    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxpk\" (UniqueName: \"kubernetes.io/projected/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-kube-api-access-ksxpk\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.228780 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.228762    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4xr\" (UniqueName: \"kubernetes.io/projected/963ec62f-272a-414d-9542-92273189e836-kube-api-access-wv4xr\") pod \"service-ca-5fffc97bdf-s7t7g\" (UID: \"963ec62f-272a-414d-9542-92273189e836\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s7t7g"
Oct 30 12:32:23.230012 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.229997    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhzw\" (UniqueName: \"kubernetes.io/projected/a3c865c9-11ec-4356-b33c-b0e4d2e6a401-kube-api-access-djhzw\") pod \"network-check-target-cqc2z\" (UID: \"a3c865c9-11ec-4356-b33c-b0e4d2e6a401\") " pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:23.230361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.230338    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nznhb\" (UniqueName: \"kubernetes.io/projected/f35180a3-5aa1-4885-a824-ea3f40d6546a-kube-api-access-nznhb\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.233425 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.233409    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7dm\" (UniqueName: \"kubernetes.io/projected/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kube-api-access-7w7dm\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.233526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.233510    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qssl\" (UniqueName: \"kubernetes.io/projected/7424da73-f41f-433c-9bae-1402110b3e6d-kube-api-access-9qssl\") pod \"ingress-canary-cjg4w\" (UID: \"7424da73-f41f-433c-9bae-1402110b3e6d\") " pod="openshift-ingress-canary/ingress-canary-cjg4w"
Oct 30 12:32:23.234906 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.234887    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztsmb\" (UniqueName: \"kubernetes.io/projected/4825a637-bc1a-4b66-acd1-c0081284ae23-kube-api-access-ztsmb\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:32:23.236228 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.236212    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxsh\" (UniqueName: \"kubernetes.io/projected/f2102d05-9eb2-4641-a057-7b02ae7d88bd-kube-api-access-rtxsh\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:23.301979 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.301913    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-netns\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.301979 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.301939    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-hostroot\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.301979 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.301957    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-netd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.301979 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.301978    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-system-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.301992    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-k8s-cni-cncf-io\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302006    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-kubelet\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302024    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-kubelet-config\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302030    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-hostroot\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302046    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-registration-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302063    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-system-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302075    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-etc-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302079    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-k8s-cni-cncf-io\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302090    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-kubelet\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302045    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-netd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302108    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-etc-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302106    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-registration-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302108    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302131    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-netns\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302146    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-dbus\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302148    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302163    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f35180a3-5aa1-4885-a824-ea3f40d6546a-crio-socket\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302190    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302212    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-multus-certs\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302222    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f35180a3-5aa1-4885-a824-ea3f40d6546a-crio-socket\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.302222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302111    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-kubelet-config\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302240    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302256    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-host\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302226    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-host\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302303    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-multus\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302318    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3349844-7e31-4934-8c31-2a208adbf1a5-dbus\") pod \"global-pull-secret-syncer-6h57s\" (UID: \"a3349844-7e31-4934-8c31-2a208adbf1a5\") " pod="kube-system/global-pull-secret-syncer-6h57s"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302259    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-run-multus-certs\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302323    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-etc-kubernetes\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302354    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-multus\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302363    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-slash\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302387    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-kubernetes\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302401    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-systemd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302365    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-etc-kubernetes\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302425    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-systemd\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302432    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-cnibin\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302452    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-kubernetes\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302459    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-slash\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302460    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302495    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-cnibin\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302497    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-var-lib-kubelet\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302505    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302527    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302539    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-var-lib-kubelet\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302553    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-socket-dir-parent\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302575    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302590    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302610    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-system-cni-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302642    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-socket-dir-parent\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302656    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-conf-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302670    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a32307-3434-4dff-91d1-8dd3e6c8369e-system-cni-dir\") pod \"multus-additional-cni-plugins-fpqzd\" (UID: \"37a32307-3434-4dff-91d1-8dd3e6c8369e\") " pod="openshift-multus/multus-additional-cni-plugins-fpqzd"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302678    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-socket-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.302670    4429 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.302686    4429 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302711    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-kubelet\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302715    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302733    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-log-socket\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302731    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-conf-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.302763    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls podName:7208585d-b1f1-4d3f-9348-2e4abaaa8e26 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:23.802740459 +0000 UTC m=+1.858234265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls") pod "dns-default-fj9wk" (UID: "7208585d-b1f1-4d3f-9348-2e4abaaa8e26") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302761    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-kubelet\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302770    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-socket-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302773    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-log-socket\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.302796    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:32:23.802780744 +0000 UTC m=+1.858274546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : secret "metrics-daemon-secret" not found
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302816    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302843    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-run\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302867    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-var-lib-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302873    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302901    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-var-lib-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302907    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc2566d-5249-44ca-be3a-697d3eb4d603-host-slash\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302916    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-run\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302935    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-modprobe-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302975    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bc2566d-5249-44ca-be3a-697d3eb4d603-host-slash\") pod \"iptables-alerter-qmdgt\" (UID: \"4bc2566d-5249-44ca-be3a-697d3eb4d603\") " pod="openshift-network-operator/iptables-alerter-qmdgt"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.302984    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-systemd\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303015    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-netns\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.303013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303037    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303057    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-modprobe-d\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303087    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-systemd\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303081    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-run-netns\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303101    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-multus-cni-dir\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303106    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-sys\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303142    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-sys-fs\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303165    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-ovn\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303176    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-sys\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303189    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cnibin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303206    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-sys-fs\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303210    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-conf\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303218    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-ovn\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303246    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4825a637-bc1a-4b66-acd1-c0081284ae23-hosts-file\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303249    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-cnibin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303261    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-bin\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303278    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-bin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303314    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4825a637-bc1a-4b66-acd1-c0081284ae23-hosts-file\") pod \"node-resolver-fttst\" (UID: \"4825a637-bc1a-4b66-acd1-c0081284ae23\") " pod="openshift-dns/node-resolver-fttst"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303321    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf513a7-048e-42d8-98fb-5b7adbc09011-host\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303336    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-cni-bin\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303346    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303362    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e07537a-35f5-40ad-b6fe-6b0c92717cf4-host-var-lib-cni-bin\") pod \"multus-k4c5j\" (UID: \"8e07537a-35f5-40ad-b6fe-6b0c92717cf4\") " pod="openshift-multus/multus-k4c5j"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303369    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysconfig\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303367    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysctl-conf\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303378    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf513a7-048e-42d8-98fb-5b7adbc09011-host\") pod \"node-ca-r8lf8\" (UID: \"6bf513a7-048e-42d8-98fb-5b7adbc09011\") " pod="openshift-image-registry/node-ca-r8lf8"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303393    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-device-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.303454    4429 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303485    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303488    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1f03eb8-15da-4d44-bc9c-8e043931b8ee-device-dir\") pod \"aws-ebs-csi-driver-node-jjzrn\" (UID: \"f1f03eb8-15da-4d44-bc9c-8e043931b8ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjzrn"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.303506    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:23.803491174 +0000 UTC m=+1.858984994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303515    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-etc-sysconfig\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303526    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-run-openvswitch\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303543    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303587    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-lib-modules\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303609    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-systemd-units\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303623    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-node-log\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303623    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303680    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-node-log\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303683    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9729abbf-37cc-43cb-bbf9-8c08a18eb37a-systemd-units\") pod \"ovnkube-node-2s9ms\" (UID: \"9729abbf-37cc-43cb-bbf9-8c08a18eb37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:23.304610 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.303715    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adf5def0-890c-405c-9706-664675f53547-lib-modules\") pod \"tuned-lgqrc\" (UID: \"adf5def0-890c-405c-9706-664675f53547\") " pod="openshift-cluster-node-tuning-operator/tuned-lgqrc"
Oct 30 12:32:23.807077 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.807036    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:23.807260 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.807089    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:23.807260 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:23.807136    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:23.807260 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.807209    4429 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 12:32:23.807260 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.807229    4429 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:23.807260 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.807247    4429 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:23.807514 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.807309    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls podName:7208585d-b1f1-4d3f-9348-2e4abaaa8e26 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.807268322 +0000 UTC m=+2.862762128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls") pod "dns-default-fj9wk" (UID: "7208585d-b1f1-4d3f-9348-2e4abaaa8e26") : secret "dns-default-metrics-tls" not found
Oct 30 12:32:23.807514 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.807333    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.807321229 +0000 UTC m=+2.862815037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : secret "metrics-daemon-secret" not found
Oct 30 12:32:23.807514 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:23.807349    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:24.807340359 +0000 UTC m=+2.862834162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:24.814253 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:24.814215    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:24.814253 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:24.814258    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:24.814747 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:24.814320    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:24.814747 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:24.814431    4429 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 12:32:24.814747 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:24.814495    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs podName:f2102d05-9eb2-4641-a057-7b02ae7d88bd nodeName:}" failed. No retries permitted until 2026-10-30 12:32:26.814481497 +0000 UTC m=+4.869975299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs") pod "network-metrics-daemon-kfgnl" (UID: "f2102d05-9eb2-4641-a057-7b02ae7d88bd") : secret "metrics-daemon-secret" not found
Oct 30 12:32:24.814747 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:24.814431    4429 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:24.814747 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:24.814593    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls podName:f35180a3-5aa1-4885-a824-ea3f40d6546a nodeName:}" failed. No retries permitted until 2026-10-30 12:32:26.814572182 +0000 UTC m=+4.870065988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnl9r" (UID: "f35180a3-5aa1-4885-a824-ea3f40d6546a") : secret "insights-runtime-extractor-tls" not found
Oct 30 12:32:24.814747 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:24.814643    4429 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:32:24.817401 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:24.817364    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7208585d-b1f1-4d3f-9348-2e4abaaa8e26-metrics-tls\") pod \"dns-default-fj9wk\" (UID: \"7208585d-b1f1-4d3f-9348-2e4abaaa8e26\") " pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:24.877180 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:24.877135    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:25.213011 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:25.212992    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fj9wk"]
Oct 30 12:32:25.214960 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:25.214925    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7208585d_b1f1_4d3f_9348_2e4abaaa8e26.slice/crio-de69c068ea41cadf5d7ab0682768cf2523d256f70a37ba17638e77c9b7a1d249 WatchSource:0}: Error finding container de69c068ea41cadf5d7ab0682768cf2523d256f70a37ba17638e77c9b7a1d249: Status 404 returned error can't find the container with id de69c068ea41cadf5d7ab0682768cf2523d256f70a37ba17638e77c9b7a1d249
Oct 30 12:32:25.217518 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:25.217502    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:32:26.113307 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:26.113216    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fj9wk" event={"ID":"7208585d-b1f1-4d3f-9348-2e4abaaa8e26","Type":"ContainerStarted","Data":"de69c068ea41cadf5d7ab0682768cf2523d256f70a37ba17638e77c9b7a1d249"}
Oct 30 12:32:26.825270 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:26.825229    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:26.825478 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:26.825338    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:26.828136 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:26.828112    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2102d05-9eb2-4641-a057-7b02ae7d88bd-metrics-certs\") pod \"network-metrics-daemon-kfgnl\" (UID: \"f2102d05-9eb2-4641-a057-7b02ae7d88bd\") " pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:26.828225 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:26.828127    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f35180a3-5aa1-4885-a824-ea3f40d6546a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnl9r\" (UID: \"f35180a3-5aa1-4885-a824-ea3f40d6546a\") " pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:26.973470 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:26.973387    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kfgnl"
Oct 30 12:32:26.977297 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:26.977264    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnl9r"
Oct 30 12:32:27.117695 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:27.117663    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fj9wk" event={"ID":"7208585d-b1f1-4d3f-9348-2e4abaaa8e26","Type":"ContainerStarted","Data":"afeafc8193e1ec34fd68e952a40dfb9d0c4c2133926db81849c835bc98e4d970"}
Oct 30 12:32:27.118050 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:27.117716    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fj9wk" event={"ID":"7208585d-b1f1-4d3f-9348-2e4abaaa8e26","Type":"ContainerStarted","Data":"488caca2cce2f605fe36d6e5bb98cd5a25cf908afcce6abcd8cc8a708fa79bc0"}
Oct 30 12:32:27.271810 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:27.271785    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kfgnl"]
Oct 30 12:32:27.273465 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:27.273437    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2102d05_9eb2_4641_a057_7b02ae7d88bd.slice/crio-67e1f2723d5327cd7fa58a1efe5d97f4e571766ec6d5251b61aff34f5c3fc3b0 WatchSource:0}: Error finding container 67e1f2723d5327cd7fa58a1efe5d97f4e571766ec6d5251b61aff34f5c3fc3b0: Status 404 returned error can't find the container with id 67e1f2723d5327cd7fa58a1efe5d97f4e571766ec6d5251b61aff34f5c3fc3b0
Oct 30 12:32:27.282585 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:27.282562    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:27.282767 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:27.282749    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnl9r"]
Oct 30 12:32:27.544338 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:27.544248    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:28.122662 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:28.122626    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnl9r" event={"ID":"f35180a3-5aa1-4885-a824-ea3f40d6546a","Type":"ContainerStarted","Data":"aa7a9cd679fd3fffe1d09078efab3b85cd9ca451df41dd3c59297c7d96ac50ed"}
Oct 30 12:32:28.123056 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:28.122686    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnl9r" event={"ID":"f35180a3-5aa1-4885-a824-ea3f40d6546a","Type":"ContainerStarted","Data":"d488fafd75a812e7fd28bc12f22598cf8b12dcc7d0f2dc9270b2ebe5a19d9f21"}
Oct 30 12:32:28.123056 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:28.122710    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnl9r" event={"ID":"f35180a3-5aa1-4885-a824-ea3f40d6546a","Type":"ContainerStarted","Data":"dbad20063bd45d8691b3a290987ec8636b7886b5f902fb2c8b6111f8c8bfce2d"}
Oct 30 12:32:28.123885 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:28.123855    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfgnl" event={"ID":"f2102d05-9eb2-4641-a057-7b02ae7d88bd","Type":"ContainerStarted","Data":"67e1f2723d5327cd7fa58a1efe5d97f4e571766ec6d5251b61aff34f5c3fc3b0"}
Oct 30 12:32:29.130240 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:29.130198    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfgnl" event={"ID":"f2102d05-9eb2-4641-a057-7b02ae7d88bd","Type":"ContainerStarted","Data":"2d91fb0062133a1e46cc863fde2317b1f70c2b4036695196d69c2533878541d1"}
Oct 30 12:32:29.130712 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:29.130266    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kfgnl" event={"ID":"f2102d05-9eb2-4641-a057-7b02ae7d88bd","Type":"ContainerStarted","Data":"2bd559adc8867f459145bb8c4f9e0751c151c115a32a5b13d3c7744de87b6827"}
Oct 30 12:32:29.797749 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:29.797715    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:29.798001 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:29.797985    4429 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:32:30.134686 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:30.134601    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnl9r" event={"ID":"f35180a3-5aa1-4885-a824-ea3f40d6546a","Type":"ContainerStarted","Data":"224435d2f32dcce5197b76cfe65b411e92896350b2c3c3abaa35c9d85c73f2b6"}
Oct 30 12:32:31.070442 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.070413    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"]
Oct 30 12:32:31.073151 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.073135    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"
Oct 30 12:32:31.094027 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.094002    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kwndj\""
Oct 30 12:32:31.108350 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.108322    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 12:32:31.108845 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.108827    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"]
Oct 30 12:32:31.152071 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.152039    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/28d28355-8803-47b6-9cdd-106503274ebc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8mtss\" (UID: \"28d28355-8803-47b6-9cdd-106503274ebc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"
Oct 30 12:32:31.171760 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.171736    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67d678455d-64qcq"]
Oct 30 12:32:31.174465 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.174448    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.194859 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.194837    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 12:32:31.208053 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.208033    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 12:32:31.208156 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.208068    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 12:32:31.208156 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.208106    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtx2d\""
Oct 30 12:32:31.223474 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.223440    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 12:32:31.224255 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.224235    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67d678455d-64qcq"]
Oct 30 12:32:31.252674 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.252652    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-trusted-ca\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.252772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.252682    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-image-registry-private-configuration\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.252772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.252700    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-installation-pull-secrets\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.252772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.252726    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86h2z\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-kube-api-access-86h2z\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.252878 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.252833    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-ca-trust-extracted\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.252878 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.252857    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-bound-sa-token\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.252939 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.252933    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-registry-tls\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.253002 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.252991    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/28d28355-8803-47b6-9cdd-106503274ebc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8mtss\" (UID: \"28d28355-8803-47b6-9cdd-106503274ebc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"
Oct 30 12:32:31.253051 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.253013    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-registry-certificates\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.255564 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.255538    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/28d28355-8803-47b6-9cdd-106503274ebc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8mtss\" (UID: \"28d28355-8803-47b6-9cdd-106503274ebc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"
Oct 30 12:32:31.353622 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.353550    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-registry-certificates\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.353622 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.353583    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-trusted-ca\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.353622 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.353607    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-image-registry-private-configuration\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.353859 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.353636    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-installation-pull-secrets\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.353859 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.353660    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86h2z\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-kube-api-access-86h2z\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.353859 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.353791    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-ca-trust-extracted\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.353859 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.353834    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-bound-sa-token\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.354053 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.353879    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-registry-tls\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.354250 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.354228    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-ca-trust-extracted\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.354616 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.354412    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-registry-certificates\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.354938 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.354915    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-trusted-ca\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.356172 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.356151    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-image-registry-private-configuration\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.356301 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.356264    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-installation-pull-secrets\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.356655 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.356637    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-registry-tls\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.378847 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.378826    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86h2z\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-kube-api-access-86h2z\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.380048 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.380028    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83-bound-sa-token\") pod \"image-registry-67d678455d-64qcq\" (UID: \"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83\") " pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.384958 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.384940    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"
Oct 30 12:32:31.485238 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.485209    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:31.681733 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.681707    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"]
Oct 30 12:32:31.683223 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:31.683197    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d28355_8803_47b6_9cdd_106503274ebc.slice/crio-bcf22abf73ca72ed7c0e94d778418ebdb03a124015e0d96c437bc86c6fea2191 WatchSource:0}: Error finding container bcf22abf73ca72ed7c0e94d778418ebdb03a124015e0d96c437bc86c6fea2191: Status 404 returned error can't find the container with id bcf22abf73ca72ed7c0e94d778418ebdb03a124015e0d96c437bc86c6fea2191
Oct 30 12:32:31.789770 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.789741    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67d678455d-64qcq"]
Oct 30 12:32:31.791672 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:31.791645    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1fb865_d9e1_4a30_9241_4c0cc7c0ae83.slice/crio-b9dda7a22423bc0bfa4b88513abe50f5121337f991d89e4507ee1de69aea26fd WatchSource:0}: Error finding container b9dda7a22423bc0bfa4b88513abe50f5121337f991d89e4507ee1de69aea26fd: Status 404 returned error can't find the container with id b9dda7a22423bc0bfa4b88513abe50f5121337f991d89e4507ee1de69aea26fd
Oct 30 12:32:31.845590 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:31.845571    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:32.144232 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:32.144195    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d678455d-64qcq" event={"ID":"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83","Type":"ContainerStarted","Data":"e2bfdc82ce79b7f7f373fd4c74fb1ba179506464b9f0a5d4c21b8401d428cc0b"}
Oct 30 12:32:32.144436 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:32.144252    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d678455d-64qcq" event={"ID":"ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83","Type":"ContainerStarted","Data":"b9dda7a22423bc0bfa4b88513abe50f5121337f991d89e4507ee1de69aea26fd"}
Oct 30 12:32:32.147043 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:32.147007    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss" event={"ID":"28d28355-8803-47b6-9cdd-106503274ebc","Type":"ContainerStarted","Data":"bcf22abf73ca72ed7c0e94d778418ebdb03a124015e0d96c437bc86c6fea2191"}
Oct 30 12:32:32.194728 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:32.194674    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67d678455d-64qcq" podStartSLOduration=1.1946569949999999 podStartE2EDuration="1.194656995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:32.194332306 +0000 UTC m=+10.249826132" watchObservedRunningTime="2026-10-30 12:32:32.194656995 +0000 UTC m=+10.250150817"
Oct 30 12:32:33.151595 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:33.151568    4429 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:32:33.151595 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:33.151563    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss" event={"ID":"28d28355-8803-47b6-9cdd-106503274ebc","Type":"ContainerStarted","Data":"b887a9854b9218acbfd49ddd72ff91b52d69cd8affdaa84b2c369d3c54189bf8"}
Oct 30 12:32:33.151897 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:33.151875    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"
Oct 30 12:32:33.156313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:33.156294    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss"
Oct 30 12:32:33.195115 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:33.195070    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8mtss" podStartSLOduration=1.375097687 podStartE2EDuration="2.195060174s" totalImagesPullingTime="819.962487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:33.194385389 +0000 UTC m=+11.249879245" watchObservedRunningTime="2026-10-30 12:32:33.195060174 +0000 UTC m=+11.250554025"
Oct 30 12:32:35.944751 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:35.944699    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:38.132506 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.132476    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fj9wk"
Oct 30 12:32:38.868377 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.868345    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"]
Oct 30 12:32:38.877796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.877767    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:38.885403 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.885384    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"]
Oct 30 12:32:38.888474 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.888460    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:38.913423 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.913396    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:32:38.913584 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.913409    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jc8s5\""
Oct 30 12:32:38.913584 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.913492    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:32:38.913967 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.913943    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:32:38.914191 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.914042    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:32:38.914829 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.914802    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"]
Oct 30 12:32:38.914829 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.914827    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 12:32:38.927213 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.927194    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 12:32:38.928088 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.928065    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pq7wj"]
Oct 30 12:32:38.931256 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.931239    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"]
Oct 30 12:32:38.931377 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.931361    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:38.932825 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.932806    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 12:32:38.932825 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.932805    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sw2b7\""
Oct 30 12:32:38.933385 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.933370    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:32:38.999547 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.999517    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f8qlc\""
Oct 30 12:32:38.999678 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.999596    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:32:38.999678 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.999612    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:32:38.999678 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:38.999637    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:32:39.008496 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008476    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-wtmp\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.008552 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008501    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxn8r\" (UniqueName: \"kubernetes.io/projected/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-api-access-lxn8r\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.008552 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008523    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05a06931-11e7-4665-850e-4d6732c39faa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.008552 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008542    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c8a7124-855e-46a2-99fd-0eba3204c556-metrics-client-ca\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.008668 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008557    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05a06931-11e7-4665-850e-4d6732c39faa-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.008668 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008605    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.008668 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008660    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.008800 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008770    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db29s\" (UniqueName: \"kubernetes.io/projected/05a06931-11e7-4665-850e-4d6732c39faa-kube-api-access-db29s\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.008850 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008809    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05a06931-11e7-4665-850e-4d6732c39faa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.008850 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008837    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-sys\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.008942 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008864    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.008942 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008900    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-textfile\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.009030 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.008948    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/093ae6a9-190d-43f8-a883-8807b82d47c5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.009082 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.009035    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-tls\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.009129 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.009081    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4kj\" (UniqueName: \"kubernetes.io/projected/9c8a7124-855e-46a2-99fd-0eba3204c556-kube-api-access-dk4kj\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.009165 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.009123    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/093ae6a9-190d-43f8-a883-8807b82d47c5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.009165 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.009142    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-root\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.009165 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.009158    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-accelerators-collector-config\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.009259 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.009175    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.110435 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110402    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-textfile\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110435 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110438    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/093ae6a9-190d-43f8-a883-8807b82d47c5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.110647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110463    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-tls\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110502    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4kj\" (UniqueName: \"kubernetes.io/projected/9c8a7124-855e-46a2-99fd-0eba3204c556-kube-api-access-dk4kj\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110529    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/093ae6a9-190d-43f8-a883-8807b82d47c5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.110647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110557    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-root\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110584    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-accelerators-collector-config\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110609    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.110647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110638    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-wtmp\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110662    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxn8r\" (UniqueName: \"kubernetes.io/projected/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-api-access-lxn8r\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110667    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-root\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110716    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05a06931-11e7-4665-850e-4d6732c39faa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110753    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c8a7124-855e-46a2-99fd-0eba3204c556-metrics-client-ca\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110780    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05a06931-11e7-4665-850e-4d6732c39faa-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110806    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110833    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110849    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-textfile\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110872    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db29s\" (UniqueName: \"kubernetes.io/projected/05a06931-11e7-4665-850e-4d6732c39faa-kube-api-access-db29s\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110902    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05a06931-11e7-4665-850e-4d6732c39faa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.110974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.110943    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/093ae6a9-190d-43f8-a883-8807b82d47c5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.111696 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.111243    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/093ae6a9-190d-43f8-a883-8807b82d47c5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.111696 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.111358    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-accelerators-collector-config\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.111696 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.111415    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-wtmp\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.111857 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.111778    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-sys\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.111857 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.111814    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.111959 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.111867    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c8a7124-855e-46a2-99fd-0eba3204c556-metrics-client-ca\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.111959 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.111936    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c8a7124-855e-46a2-99fd-0eba3204c556-sys\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.112062 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.111988    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.112265 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.112240    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05a06931-11e7-4665-850e-4d6732c39faa-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.113486 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.113462    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-tls\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.113600 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.113549    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05a06931-11e7-4665-850e-4d6732c39faa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.113941 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.113915    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.114197 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.114181    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05a06931-11e7-4665-850e-4d6732c39faa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.114503 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.114487    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c8a7124-855e-46a2-99fd-0eba3204c556-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.114735 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.114713    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.136076 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.136027    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4kj\" (UniqueName: \"kubernetes.io/projected/9c8a7124-855e-46a2-99fd-0eba3204c556-kube-api-access-dk4kj\") pod \"node-exporter-pq7wj\" (UID: \"9c8a7124-855e-46a2-99fd-0eba3204c556\") " pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.136401 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.136257    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxn8r\" (UniqueName: \"kubernetes.io/projected/093ae6a9-190d-43f8-a883-8807b82d47c5-kube-api-access-lxn8r\") pod \"kube-state-metrics-5dc8d7d89b-69s5m\" (UID: \"093ae6a9-190d-43f8-a883-8807b82d47c5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.137090 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.137074    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db29s\" (UniqueName: \"kubernetes.io/projected/05a06931-11e7-4665-850e-4d6732c39faa-kube-api-access-db29s\") pod \"openshift-state-metrics-7bd7489957-6qvps\" (UID: \"05a06931-11e7-4665-850e-4d6732c39faa\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.194486 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.194459    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"
Oct 30 12:32:39.203720 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.203696    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"
Oct 30 12:32:39.247705 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.247681    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pq7wj"
Oct 30 12:32:39.273618 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:39.273589    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8a7124_855e_46a2_99fd_0eba3204c556.slice/crio-2aaa62a3a77cf3649c93333c1a407020f778690597e716d2353376cf0556af5f WatchSource:0}: Error finding container 2aaa62a3a77cf3649c93333c1a407020f778690597e716d2353376cf0556af5f: Status 404 returned error can't find the container with id 2aaa62a3a77cf3649c93333c1a407020f778690597e716d2353376cf0556af5f
Oct 30 12:32:39.508041 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.508016    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m"]
Oct 30 12:32:39.508737 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:39.508709    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093ae6a9_190d_43f8_a883_8807b82d47c5.slice/crio-5205bed525f9b0c08d26015787ac879eeb35d216f3f265c411349e2da5ccc8b5 WatchSource:0}: Error finding container 5205bed525f9b0c08d26015787ac879eeb35d216f3f265c411349e2da5ccc8b5: Status 404 returned error can't find the container with id 5205bed525f9b0c08d26015787ac879eeb35d216f3f265c411349e2da5ccc8b5
Oct 30 12:32:39.514450 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:39.512410    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps"]
Oct 30 12:32:39.517014 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:39.516987    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a06931_11e7_4665_850e_4d6732c39faa.slice/crio-c257507b662465818e68845a20ff913926e7e429363994bf2fdbc2cb5efa6339 WatchSource:0}: Error finding container c257507b662465818e68845a20ff913926e7e429363994bf2fdbc2cb5efa6339: Status 404 returned error can't find the container with id c257507b662465818e68845a20ff913926e7e429363994bf2fdbc2cb5efa6339
Oct 30 12:32:40.182710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:40.182669    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pq7wj" event={"ID":"9c8a7124-855e-46a2-99fd-0eba3204c556","Type":"ContainerStarted","Data":"2aaa62a3a77cf3649c93333c1a407020f778690597e716d2353376cf0556af5f"}
Oct 30 12:32:40.184749 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:40.184712    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps" event={"ID":"05a06931-11e7-4665-850e-4d6732c39faa","Type":"ContainerStarted","Data":"994627dcf5eb8e7fbc2711776a273021924d64205663f5a0873790f1b61c7576"}
Oct 30 12:32:40.184889 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:40.184771    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps" event={"ID":"05a06931-11e7-4665-850e-4d6732c39faa","Type":"ContainerStarted","Data":"123a14a9cb53f647f66f02d91bd0edfc06505e5f3336cb2a4c1dd3066b7d20f5"}
Oct 30 12:32:40.184889 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:40.184799    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps" event={"ID":"05a06931-11e7-4665-850e-4d6732c39faa","Type":"ContainerStarted","Data":"c257507b662465818e68845a20ff913926e7e429363994bf2fdbc2cb5efa6339"}
Oct 30 12:32:40.186428 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:40.186360    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m" event={"ID":"093ae6a9-190d-43f8-a883-8807b82d47c5","Type":"ContainerStarted","Data":"5205bed525f9b0c08d26015787ac879eeb35d216f3f265c411349e2da5ccc8b5"}
Oct 30 12:32:41.192793 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:41.192754    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m" event={"ID":"093ae6a9-190d-43f8-a883-8807b82d47c5","Type":"ContainerStarted","Data":"5234a1b7312297ff5a03c205497dfe2466644ed7c9f020ebfae1915107b9356d"}
Oct 30 12:32:41.193199 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:41.192818    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m" event={"ID":"093ae6a9-190d-43f8-a883-8807b82d47c5","Type":"ContainerStarted","Data":"766971536e2e12d38421828964533b97cea1f965c1626512c49b98aa3a2cf814"}
Oct 30 12:32:41.193199 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:41.192849    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m" event={"ID":"093ae6a9-190d-43f8-a883-8807b82d47c5","Type":"ContainerStarted","Data":"685ab9961b8c9863345880e1eb3a0bdf6be1c476bf55a7e6711b8b5f4e4b791e"}
Oct 30 12:32:41.194575 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:41.194549    4429 generic.go:356] "Generic (PLEG): container finished" podID="9c8a7124-855e-46a2-99fd-0eba3204c556" containerID="1fdefa8de17a3684111c24889e66aa4362a25cb0381c110e7296013cc30ec1c9" exitCode=0
Oct 30 12:32:41.194653 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:41.194630    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pq7wj" event={"ID":"9c8a7124-855e-46a2-99fd-0eba3204c556","Type":"ContainerDied","Data":"1fdefa8de17a3684111c24889e66aa4362a25cb0381c110e7296013cc30ec1c9"}
Oct 30 12:32:41.196895 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:41.196873    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps" event={"ID":"05a06931-11e7-4665-850e-4d6732c39faa","Type":"ContainerStarted","Data":"202cf937397ab9d7b04419a564c6fa8c3fe20950185099352b917107aa6a079d"}
Oct 30 12:32:41.244885 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:41.244838    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-69s5m" podStartSLOduration=2.043870094 podStartE2EDuration="3.244824634s" totalImagesPullingTime="1.20095454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:41.244444018 +0000 UTC m=+19.299937843" watchObservedRunningTime="2026-10-30 12:32:41.244824634 +0000 UTC m=+19.300318458"
Oct 30 12:32:41.338203 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:41.338159    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6qvps" podStartSLOduration=2.369510941 podStartE2EDuration="3.338146043s" totalImagesPullingTime="968.635102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:41.337425491 +0000 UTC m=+19.392919315" watchObservedRunningTime="2026-10-30 12:32:41.338146043 +0000 UTC m=+19.393639868"
Oct 30 12:32:42.202047 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:42.202012    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pq7wj" event={"ID":"9c8a7124-855e-46a2-99fd-0eba3204c556","Type":"ContainerStarted","Data":"cad66d02d3f24f3caafdaf958ed4b5bc23beffe2cbcc97ce129bb1311de51576"}
Oct 30 12:32:42.202574 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:42.202074    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pq7wj" event={"ID":"9c8a7124-855e-46a2-99fd-0eba3204c556","Type":"ContainerStarted","Data":"cc92c7209f9186410daced7e1696c48ae358d6d2e1bab088ce5654a6f2458291"}
Oct 30 12:32:42.283176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:42.283127    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pq7wj" podStartSLOduration=3.16680684 podStartE2EDuration="4.283114215s" totalImagesPullingTime="1.024891879s" totalInitContainerRuntime="91.415496ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:42.282820409 +0000 UTC m=+20.338314256" watchObservedRunningTime="2026-10-30 12:32:42.283114215 +0000 UTC m=+20.338608039"
Oct 30 12:32:42.373260 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:32:42.371612    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:32:43.379620 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.379591    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57f9c5fc79-snqqs"]
Oct 30 12:32:43.382897 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.382874    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.400776 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.400750    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 12:32:43.400924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.400810    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 12:32:43.414641 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.414616    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 12:32:43.414758 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.414680    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ecvajmn2s8icq\""
Oct 30 12:32:43.414758 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.414712    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 12:32:43.414852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.414772    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-66vrv\""
Oct 30 12:32:43.415881 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.415860    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57f9c5fc79-snqqs"]
Oct 30 12:32:43.546834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.546801    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067d351d-52ac-4f35-809f-99c8e9acf213-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.546834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.546838    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggs5\" (UniqueName: \"kubernetes.io/projected/067d351d-52ac-4f35-809f-99c8e9acf213-kube-api-access-wggs5\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.547037 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.546860    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-secret-metrics-server-client-certs\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.547037 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.546981    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-secret-metrics-server-tls\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.547037 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.547009    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/067d351d-52ac-4f35-809f-99c8e9acf213-metrics-server-audit-profiles\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.547037 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.547028    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-client-ca-bundle\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.547154 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.547046    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/067d351d-52ac-4f35-809f-99c8e9acf213-audit-log\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.647905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.647807    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067d351d-52ac-4f35-809f-99c8e9acf213-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.647905 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.647859    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggs5\" (UniqueName: \"kubernetes.io/projected/067d351d-52ac-4f35-809f-99c8e9acf213-kube-api-access-wggs5\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.648088 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.647973    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-secret-metrics-server-client-certs\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.648088 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.648029    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-secret-metrics-server-tls\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.648088 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.648056    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/067d351d-52ac-4f35-809f-99c8e9acf213-metrics-server-audit-profiles\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.648088 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.648082    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-client-ca-bundle\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.648269 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.648112    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/067d351d-52ac-4f35-809f-99c8e9acf213-audit-log\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.648558 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.648523    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/067d351d-52ac-4f35-809f-99c8e9acf213-audit-log\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.648664 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.648638    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/067d351d-52ac-4f35-809f-99c8e9acf213-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.649375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.649348    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/067d351d-52ac-4f35-809f-99c8e9acf213-metrics-server-audit-profiles\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.650740 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.650720    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-secret-metrics-server-tls\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.650830 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.650788    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-client-ca-bundle\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.650957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.650935    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/067d351d-52ac-4f35-809f-99c8e9acf213-secret-metrics-server-client-certs\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.677197 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.677166    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggs5\" (UniqueName: \"kubernetes.io/projected/067d351d-52ac-4f35-809f-99c8e9acf213-kube-api-access-wggs5\") pod \"metrics-server-57f9c5fc79-snqqs\" (UID: \"067d351d-52ac-4f35-809f-99c8e9acf213\") " pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:43.693882 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:43.693859    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:32:44.000223 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:44.000198    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57f9c5fc79-snqqs"]
Oct 30 12:32:44.002060 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:44.002035    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067d351d_52ac_4f35_809f_99c8e9acf213.slice/crio-dd8112b36f73c7e01f4f22c4a1e0737299b1ca1ca8e952cccea1353c90fd08ab WatchSource:0}: Error finding container dd8112b36f73c7e01f4f22c4a1e0737299b1ca1ca8e952cccea1353c90fd08ab: Status 404 returned error can't find the container with id dd8112b36f73c7e01f4f22c4a1e0737299b1ca1ca8e952cccea1353c90fd08ab
Oct 30 12:32:44.209495 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:44.209463    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs" event={"ID":"067d351d-52ac-4f35-809f-99c8e9acf213","Type":"ContainerStarted","Data":"dd8112b36f73c7e01f4f22c4a1e0737299b1ca1ca8e952cccea1353c90fd08ab"}
Oct 30 12:32:45.819143 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.819108    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:32:45.822947 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.822926    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.841112 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.841082    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 12:32:45.854935 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.854911    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 12:32:45.855066 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.854932    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5ai462sh5d18p\""
Oct 30 12:32:45.855066 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.854951    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 12:32:45.855066 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.854969    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 12:32:45.855212 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.854986    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 12:32:45.855212 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.855086    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zklcb\""
Oct 30 12:32:45.855212 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.854998    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 12:32:45.855212 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.854991    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 12:32:45.855893 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.855874    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:32:45.941961 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.941933    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 12:32:45.942252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.942005    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 12:32:45.942641 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.942547    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 12:32:45.942641 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.942574    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 12:32:45.942641 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.942575    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 12:32:45.957034 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.957010    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 12:32:45.957034 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.957009    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 12:32:45.968202 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968178    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968267 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968207    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968267 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968231    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968267 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968262    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968415 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968386    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968456 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968417    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl99k\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-kube-api-access-xl99k\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968456 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968451    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968551 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968502    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968551 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968538    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968620 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968573    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968649 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968634    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968678 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968654    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968678 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968674    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968749 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968688    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968749 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968708    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968821 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968774    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968821 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968792    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968821 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968808    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968961 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968843    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:45.968961 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:45.968905    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069575 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069501    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069575 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069550    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069575 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069567    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069594    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069619    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069640    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069667    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl99k\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-kube-api-access-xl99k\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069699    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069725    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069747    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069771    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069797    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.069816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069819    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069851    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069878    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069923    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069965    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.069993    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.070024    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.070047    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070856 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.070354    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.070856 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.070598    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.071006 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.070984    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.071224 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.071193    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.071352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.071326    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.073712 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.073679    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-web-config\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.073960 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.073703    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.074064 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.074043    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config-out\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.074175 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.074151    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.074237 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.074156    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.074854 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.074824    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.074947 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.074890    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.075407 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.075378    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.075711 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.075682    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.075809 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.075684    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.076246 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.076222    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.076779 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.076762    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.076931 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.076914    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.077075 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.077056    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.095160 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.095134    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl99k\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-kube-api-access-xl99k\") pod \"prometheus-k8s-0\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.147841 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.147813    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:32:46.218940 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.218900    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs" event={"ID":"067d351d-52ac-4f35-809f-99c8e9acf213","Type":"ContainerStarted","Data":"e6068a0e42a4376fd81469a92f73cab3ee4a6c82923b8594753f55c9b13633f4"}
Oct 30 12:32:46.265581 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.265535    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs" podStartSLOduration=2.081018968 podStartE2EDuration="3.265522977s" totalImagesPullingTime="1.184504009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:46.264857393 +0000 UTC m=+24.320351217" watchObservedRunningTime="2026-10-30 12:32:46.265522977 +0000 UTC m=+24.321016797"
Oct 30 12:32:46.483162 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:46.483133    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:32:46.485405 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:32:46.485379    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca87bda_6bd8_43ac_be3d_b1eee393f266.slice/crio-630aec91551f87228e4f7397e63f693cc83eb0f313620e88717c5815cd68e199 WatchSource:0}: Error finding container 630aec91551f87228e4f7397e63f693cc83eb0f313620e88717c5815cd68e199: Status 404 returned error can't find the container with id 630aec91551f87228e4f7397e63f693cc83eb0f313620e88717c5815cd68e199
Oct 30 12:32:47.228067 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:47.228018    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerStarted","Data":"630aec91551f87228e4f7397e63f693cc83eb0f313620e88717c5815cd68e199"}
Oct 30 12:32:47.644023 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:47.643943    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9ms"
Oct 30 12:32:48.232453 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:48.232422    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerID="54f1283755e2b45dfa67d2398803fb3d2dd8f70360eb75008deefba470aac578" exitCode=0
Oct 30 12:32:48.232898 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:48.232516    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerDied","Data":"54f1283755e2b45dfa67d2398803fb3d2dd8f70360eb75008deefba470aac578"}
Oct 30 12:32:48.456855 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:48.456828    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xz8gl"
Oct 30 12:32:51.254474 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:51.254400    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerStarted","Data":"8bfce38c2c6d5beef8d625c873ccd1693d266968acb3855f5dbc254685f350ad"}
Oct 30 12:32:51.254822 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:51.254491    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerStarted","Data":"6d17528358508d4889826a8c0c149266c463c4b8b056d9698733dd9c0b2d7a77"}
Oct 30 12:32:51.848869 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:51.848838    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cqc2z"
Oct 30 12:32:53.265177 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:53.265143    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerStarted","Data":"9f8867ca6fcfa9a762d832b9e09d6dea71e773c2b0abecc7a66f1d51a5b9d90e"}
Oct 30 12:32:53.265672 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:53.265232    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerStarted","Data":"badbaa480298fa2c558d88fe5b0ab14a680958773cfb18ca71309cfbfebd72c8"}
Oct 30 12:32:53.265672 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:53.265258    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerStarted","Data":"536d3e634f381b02340cf6d329c7d22284843403131f30245a8c766867d07463"}
Oct 30 12:32:53.265672 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:53.265315    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerStarted","Data":"b07df4fcc62219bfff7d1e043e7fb0f0c9b5f4dd8bffd22662ca8f8b4e60731b"}
Oct 30 12:32:53.338540 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:53.338493    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.755039698 podStartE2EDuration="8.33847901s" totalImagesPullingTime="5.568805585s" totalInitContainerRuntime="14.633727ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:53.337357993 +0000 UTC m=+31.392851835" watchObservedRunningTime="2026-10-30 12:32:53.33847901 +0000 UTC m=+31.393972835"
Oct 30 12:32:55.949797 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:55.949772    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67d678455d-64qcq"
Oct 30 12:32:56.148043 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:32:56.148009    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:03.694352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:03.694242    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:33:03.694352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:03.694336    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:33:08.224467 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:08.224440    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:08.243222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:08.243196    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:08.330539 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:08.330510    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:12.370974 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:33:12.370938    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:33:23.700146 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:23.700112    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:33:23.704023 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:23.704000    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57f9c5fc79-snqqs"
Oct 30 12:33:27.942506 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:27.942475    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cjg4w_7424da73-f41f-433c-9bae-1402110b3e6d/serve-healthcheck-canary/0.log"
Oct 30 12:33:34.138551 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.138516    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:33:34.139233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.138914    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="prometheus" containerID="cri-o://6d17528358508d4889826a8c0c149266c463c4b8b056d9698733dd9c0b2d7a77" gracePeriod=600
Oct 30 12:33:34.139233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.138973    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy" containerID="cri-o://badbaa480298fa2c558d88fe5b0ab14a680958773cfb18ca71309cfbfebd72c8" gracePeriod=600
Oct 30 12:33:34.139233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.138999    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy-web" containerID="cri-o://536d3e634f381b02340cf6d329c7d22284843403131f30245a8c766867d07463" gracePeriod=600
Oct 30 12:33:34.139233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.139028    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="config-reloader" containerID="cri-o://8bfce38c2c6d5beef8d625c873ccd1693d266968acb3855f5dbc254685f350ad" gracePeriod=600
Oct 30 12:33:34.139233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.139059    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9f8867ca6fcfa9a762d832b9e09d6dea71e773c2b0abecc7a66f1d51a5b9d90e" gracePeriod=600
Oct 30 12:33:34.139233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.138977    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="thanos-sidecar" containerID="cri-o://b07df4fcc62219bfff7d1e043e7fb0f0c9b5f4dd8bffd22662ca8f8b4e60731b" gracePeriod=600
Oct 30 12:33:34.408740 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408661    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerID="9f8867ca6fcfa9a762d832b9e09d6dea71e773c2b0abecc7a66f1d51a5b9d90e" exitCode=0
Oct 30 12:33:34.408740 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408686    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerID="badbaa480298fa2c558d88fe5b0ab14a680958773cfb18ca71309cfbfebd72c8" exitCode=0
Oct 30 12:33:34.408740 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408693    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerID="b07df4fcc62219bfff7d1e043e7fb0f0c9b5f4dd8bffd22662ca8f8b4e60731b" exitCode=0
Oct 30 12:33:34.408740 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408701    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerID="8bfce38c2c6d5beef8d625c873ccd1693d266968acb3855f5dbc254685f350ad" exitCode=0
Oct 30 12:33:34.408740 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408707    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerID="6d17528358508d4889826a8c0c149266c463c4b8b056d9698733dd9c0b2d7a77" exitCode=0
Oct 30 12:33:34.408740 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408732    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerDied","Data":"9f8867ca6fcfa9a762d832b9e09d6dea71e773c2b0abecc7a66f1d51a5b9d90e"}
Oct 30 12:33:34.409021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408821    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerDied","Data":"badbaa480298fa2c558d88fe5b0ab14a680958773cfb18ca71309cfbfebd72c8"}
Oct 30 12:33:34.409021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408849    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerDied","Data":"b07df4fcc62219bfff7d1e043e7fb0f0c9b5f4dd8bffd22662ca8f8b4e60731b"}
Oct 30 12:33:34.409021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408874    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerDied","Data":"8bfce38c2c6d5beef8d625c873ccd1693d266968acb3855f5dbc254685f350ad"}
Oct 30 12:33:34.409021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:34.408897    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerDied","Data":"6d17528358508d4889826a8c0c149266c463c4b8b056d9698733dd9c0b2d7a77"}
Oct 30 12:33:35.417397 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.417370    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerID="536d3e634f381b02340cf6d329c7d22284843403131f30245a8c766867d07463" exitCode=0
Oct 30 12:33:35.417739 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.417437    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerDied","Data":"536d3e634f381b02340cf6d329c7d22284843403131f30245a8c766867d07463"}
Oct 30 12:33:35.489680 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.489658    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:35.582634 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582551    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-serving-certs-ca-bundle\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582634 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582587    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-kube-api-access-xl99k\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-kube-api-access-xl99k\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582634 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582606    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-grpc-tls\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582637    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-trusted-ca-bundle\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582678    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-db\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582696    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-web-config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-web-config\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582717    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-metrics-client-certs\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582759    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-0\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582787    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-tls\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582815    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582854    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-2\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582895    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-1\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.582924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582922    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-kube-rbac-proxy\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582955    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-kubelet-serving-ca-bundle\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.582988    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583008    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-serving-certs-ca-bundle" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583023    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config-out\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583188    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583242    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-metrics-client-ca\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583272    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-tls-assets\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-tls-assets\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583304    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-trusted-ca-bundle" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:35.583430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583327    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-thanos-prometheus-http-client-file\") pod \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\" (UID: \"1ca87bda-6bd8-43ac-be3d-b1eee393f266\") "
Oct 30 12:33:35.584063 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583578    4429 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.584063 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.583600    4429 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.584182 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.584129    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-2" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:35.584442 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.584418    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-db" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:33:35.585075 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.585050    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-0" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:35.586969 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.586208    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-1" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:35.586969 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.586300    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-kube-api-access-xl99k" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "kube-api-access-xl99k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:35.586969 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.586759    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-metrics-client-ca" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:35.587675 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.587317    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-thanos-prometheus-http-client-file" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.588591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.588057    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-kubelet-serving-ca-bundle" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:33:35.588591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.588426    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config-out" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:33:35.588855 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.588835    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.589003 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.588845    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-thanos-sidecar-tls" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.589118 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.588908    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-tls-assets" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:33:35.589236 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.589118    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-grpc-tls" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.589457 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.589431    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-kube-rbac-proxy" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.590315 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.590261    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-tls" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.590659 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.590572    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-metrics-client-certs" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.590985 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.590947    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-kube-rbac-proxy-web" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.602761 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.602739    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-web-config" pod "1ca87bda-6bd8-43ac-be3d-b1eee393f266" (UID: "1ca87bda-6bd8-43ac-be3d-b1eee393f266"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:33:35.684157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684126    4429 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684156    4429 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684165    4429 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684175    4429 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684185    4429 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684194    4429 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-kube-rbac-proxy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684203    4429 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684213    4429 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684223    4429 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-config-out\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684231    4429 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684239    4429 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ca87bda-6bd8-43ac-be3d-b1eee393f266-configmap-metrics-client-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684247    4429 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-tls-assets\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684256    4429 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684265    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl99k\" (UniqueName: \"kubernetes.io/projected/1ca87bda-6bd8-43ac-be3d-b1eee393f266-kube-api-access-xl99k\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684273    4429 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-grpc-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684301    4429 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1ca87bda-6bd8-43ac-be3d-b1eee393f266-prometheus-k8s-db\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684309    4429 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-web-config\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:35.684382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:35.684318    4429 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1ca87bda-6bd8-43ac-be3d-b1eee393f266-secret-metrics-client-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:33:36.424082 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.424002    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1ca87bda-6bd8-43ac-be3d-b1eee393f266","Type":"ContainerDied","Data":"630aec91551f87228e4f7397e63f693cc83eb0f313620e88717c5815cd68e199"}
Oct 30 12:33:36.424082 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.424043    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.424082 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.424075    4429 scope.go:122] "RemoveContainer" containerID="9f8867ca6fcfa9a762d832b9e09d6dea71e773c2b0abecc7a66f1d51a5b9d90e"
Oct 30 12:33:36.431620 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.431601    4429 scope.go:122] "RemoveContainer" containerID="badbaa480298fa2c558d88fe5b0ab14a680958773cfb18ca71309cfbfebd72c8"
Oct 30 12:33:36.438375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.438360    4429 scope.go:122] "RemoveContainer" containerID="536d3e634f381b02340cf6d329c7d22284843403131f30245a8c766867d07463"
Oct 30 12:33:36.445220 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.445204    4429 scope.go:122] "RemoveContainer" containerID="b07df4fcc62219bfff7d1e043e7fb0f0c9b5f4dd8bffd22662ca8f8b4e60731b"
Oct 30 12:33:36.451482 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.451465    4429 scope.go:122] "RemoveContainer" containerID="8bfce38c2c6d5beef8d625c873ccd1693d266968acb3855f5dbc254685f350ad"
Oct 30 12:33:36.458314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.458295    4429 scope.go:122] "RemoveContainer" containerID="6d17528358508d4889826a8c0c149266c463c4b8b056d9698733dd9c0b2d7a77"
Oct 30 12:33:36.464999 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.464977    4429 scope.go:122] "RemoveContainer" containerID="54f1283755e2b45dfa67d2398803fb3d2dd8f70360eb75008deefba470aac578"
Oct 30 12:33:36.496619 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.496595    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:33:36.510628 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.510599    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:33:36.533823 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.533790    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:33:36.534046 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534035    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="thanos-sidecar"
Oct 30 12:33:36.534084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534048    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="thanos-sidecar"
Oct 30 12:33:36.534084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534058    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy-web"
Oct 30 12:33:36.534084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534062    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy-web"
Oct 30 12:33:36.534084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534073    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy"
Oct 30 12:33:36.534084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534077    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534091    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="prometheus"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534096    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="prometheus"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534105    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="config-reloader"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534109    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="config-reloader"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534115    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="init-config-reloader"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534120    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="init-config-reloader"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534127    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy-thanos"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534132    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy-thanos"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534172    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="config-reloader"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534178    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="thanos-sidecar"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534184    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy-web"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534189    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534195    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="prometheus"
Oct 30 12:33:36.534226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.534200    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" containerName="kube-rbac-proxy-thanos"
Oct 30 12:33:36.539254 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.539239    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.556683 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.556657    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 12:33:36.556683 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.556665    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 12:33:36.556842 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.556707    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 12:33:36.557499 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.557479    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 12:33:36.557499 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.557494    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 12:33:36.557631 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.557491    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 12:33:36.557631 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.557533    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 12:33:36.557631 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.557547    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 12:33:36.570582 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.570561    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 12:33:36.570739 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.570618    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 12:33:36.570739 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.570718    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5ai462sh5d18p\""
Oct 30 12:33:36.570739 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.570718    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zklcb\""
Oct 30 12:33:36.570739 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.570735    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 12:33:36.570953 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.570781    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 12:33:36.571775 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.571755    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 12:33:36.571867 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.571803    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 12:33:36.572683 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.572669    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:33:36.693008 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.692982    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693015    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693033    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693051    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693121    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693345 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693234    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693345 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693251    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693345 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693267    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693345 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693305    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f058f46-6d9b-4469-b091-063d2f667260-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693484 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693363    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693484 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693380    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-config\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693484 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693400    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693484 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693438    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693484 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693469    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693510    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f058f46-6d9b-4469-b091-063d2f667260-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693536    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693559    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693576    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbw9z\" (UniqueName: \"kubernetes.io/projected/5f058f46-6d9b-4469-b091-063d2f667260-kube-api-access-hbw9z\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693607    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.693781 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.693661    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794329    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794372    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794396    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794414    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794436    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794461    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f058f46-6d9b-4469-b091-063d2f667260-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794488    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794516    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-config\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794546    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794570    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794594    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794627    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f058f46-6d9b-4469-b091-063d2f667260-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794649    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794670    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.794701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794692    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbw9z\" (UniqueName: \"kubernetes.io/projected/5f058f46-6d9b-4469-b091-063d2f667260-kube-api-access-hbw9z\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.795381 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794724    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.795381 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794753    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.795381 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794780    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.795381 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794815    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.795381 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.794840    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.795381 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.795240    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.795381 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.795322    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.795749 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.795461    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.798104 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.798080    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.799412    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.799418    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.800008    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.800215    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.800374    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-config\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.800745    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.800938    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.801010    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f058f46-6d9b-4469-b091-063d2f667260-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.801553    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.801628    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.802062    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.802160    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.802226    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f058f46-6d9b-4469-b091-063d2f667260-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.804710 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.802615    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f058f46-6d9b-4469-b091-063d2f667260-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.805410 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.804911    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f058f46-6d9b-4469-b091-063d2f667260-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.821122 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.821090    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbw9z\" (UniqueName: \"kubernetes.io/projected/5f058f46-6d9b-4469-b091-063d2f667260-kube-api-access-hbw9z\") pod \"prometheus-k8s-0\" (UID: \"5f058f46-6d9b-4469-b091-063d2f667260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:36.863861 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:36.863812    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:37.193249 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:37.193229    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:33:37.194592 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:33:37.194568    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f058f46_6d9b_4469_b091_063d2f667260.slice/crio-b83b797b36162d168a8da39daf814a16d7087bef0c187d2641b737ef944392a1 WatchSource:0}: Error finding container b83b797b36162d168a8da39daf814a16d7087bef0c187d2641b737ef944392a1: Status 404 returned error can't find the container with id b83b797b36162d168a8da39daf814a16d7087bef0c187d2641b737ef944392a1
Oct 30 12:33:37.433274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:37.433236    4429 generic.go:356] "Generic (PLEG): container finished" podID="5f058f46-6d9b-4469-b091-063d2f667260" containerID="bdd4d8b11302aa833e1f641df560178268cc6afc3e757e985e64fd507acc6475" exitCode=0
Oct 30 12:33:37.433682 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:37.433329    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f058f46-6d9b-4469-b091-063d2f667260","Type":"ContainerDied","Data":"bdd4d8b11302aa833e1f641df560178268cc6afc3e757e985e64fd507acc6475"}
Oct 30 12:33:37.433682 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:37.433392    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f058f46-6d9b-4469-b091-063d2f667260","Type":"ContainerStarted","Data":"b83b797b36162d168a8da39daf814a16d7087bef0c187d2641b737ef944392a1"}
Oct 30 12:33:38.073143 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:38.073067    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca87bda-6bd8-43ac-be3d-b1eee393f266" path="/var/lib/kubelet/pods/1ca87bda-6bd8-43ac-be3d-b1eee393f266/volumes"
Oct 30 12:33:38.439802 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:38.439771    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f058f46-6d9b-4469-b091-063d2f667260","Type":"ContainerStarted","Data":"429de45b47e9fd45da4247614d033bfaed060daebf2a2310a1eb63e09f47aa83"}
Oct 30 12:33:38.440194 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:38.439834    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f058f46-6d9b-4469-b091-063d2f667260","Type":"ContainerStarted","Data":"5f54a195a8d17f0d5ca2351ab67b6966a10cd666a2a33b1ad56937f6eb5b5ef6"}
Oct 30 12:33:38.440194 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:38.439859    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f058f46-6d9b-4469-b091-063d2f667260","Type":"ContainerStarted","Data":"b95801faf799c362808b5f7cdaffdd3bdb264a34f90de8e659609bf4eb135c41"}
Oct 30 12:33:38.440194 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:38.439919    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f058f46-6d9b-4469-b091-063d2f667260","Type":"ContainerStarted","Data":"391bc23d15cec09d82d6ffaedf3a6cee5392b6db225ad6ca10ead96921c932b2"}
Oct 30 12:33:38.440194 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:38.439943    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f058f46-6d9b-4469-b091-063d2f667260","Type":"ContainerStarted","Data":"06c2c57b218b01eb0ed2b1b373993bc62c96631a00c627b6f058adbbc4e9ff53"}
Oct 30 12:33:38.440194 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:38.439968    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f058f46-6d9b-4469-b091-063d2f667260","Type":"ContainerStarted","Data":"f23d0c004a9d863cd7fd733f87e99a49e7089d23cc9c0c058019647d417528bc"}
Oct 30 12:33:38.512132 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:38.512082    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.49364677 podStartE2EDuration="2.512068224s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.421454ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:33:38.511446495 +0000 UTC m=+76.566940328" watchObservedRunningTime="2026-10-30 12:33:38.512068224 +0000 UTC m=+76.567562047"
Oct 30 12:33:41.863937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:33:41.863903    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:33:42.370071 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:33:42.370041    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:34:12.370989 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:34:12.370952    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:34:36.864603 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:34:36.864508    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:34:36.880503 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:34:36.880484    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:34:37.658600 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:34:37.658574    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:34:42.371256 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:34:42.371217    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:35:12.370515 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:35:12.370474    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:35:42.369917 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:35:42.369882    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:36:12.371003 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:36:12.370912    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:36:42.370708 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:36:42.370670    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:37:12.371422 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:37:12.371383    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:37:42.369958 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:37:42.369884    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:37:51.768132 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.768101    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"]
Oct 30 12:37:51.771419 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.771397    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:51.839235 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.839211    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"]
Oct 30 12:37:51.848370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.848351    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 12:37:51.848609 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.848589    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n948z\""
Oct 30 12:37:51.848689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.848591    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 12:37:51.848689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.848596    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 12:37:51.848689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.848635    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 12:37:51.848796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.848694    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 12:37:51.914448 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.914411    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2050faf1-dfaf-4f24-9b62-112fe6957aed-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:51.914578 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.914450    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:51.914578 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.914472    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:51.914650 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.914583    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2050faf1-dfaf-4f24-9b62-112fe6957aed-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:51.914650 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.914611    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpqv\" (UniqueName: \"kubernetes.io/projected/2050faf1-dfaf-4f24-9b62-112fe6957aed-kube-api-access-jqpqv\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:51.914650 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:51.914633    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.015786 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.015761    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2050faf1-dfaf-4f24-9b62-112fe6957aed-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.015899 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.015790    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.015899 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.015808    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.015899 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.015826    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2050faf1-dfaf-4f24-9b62-112fe6957aed-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.015899 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.015841    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpqv\" (UniqueName: \"kubernetes.io/projected/2050faf1-dfaf-4f24-9b62-112fe6957aed-kube-api-access-jqpqv\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.015899 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.015863    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.016252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.016217    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2050faf1-dfaf-4f24-9b62-112fe6957aed-manifests\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.018379 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.018328    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.018467 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.018420    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.018622 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.018604    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2050faf1-dfaf-4f24-9b62-112fe6957aed-tmp\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.018724 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.018708    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2050faf1-dfaf-4f24-9b62-112fe6957aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.049828 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.049803    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpqv\" (UniqueName: \"kubernetes.io/projected/2050faf1-dfaf-4f24-9b62-112fe6957aed-kube-api-access-jqpqv\") pod \"opendatahub-operator-controller-manager-5c9d568b4f-tftjf\" (UID: \"2050faf1-dfaf-4f24-9b62-112fe6957aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.085885 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.085855    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:52.530437 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:37:52.530409    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2050faf1_dfaf_4f24_9b62_112fe6957aed.slice/crio-bdb52786fa08abdd49003147ce2c60daf0cea18a6ef6b6beaecdc0a56430adfe WatchSource:0}: Error finding container bdb52786fa08abdd49003147ce2c60daf0cea18a6ef6b6beaecdc0a56430adfe: Status 404 returned error can't find the container with id bdb52786fa08abdd49003147ce2c60daf0cea18a6ef6b6beaecdc0a56430adfe
Oct 30 12:37:52.532052 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.532036    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:37:52.549855 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:52.549830    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"]
Oct 30 12:37:53.298051 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:53.297974    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf" event={"ID":"2050faf1-dfaf-4f24-9b62-112fe6957aed","Type":"ContainerStarted","Data":"bdb52786fa08abdd49003147ce2c60daf0cea18a6ef6b6beaecdc0a56430adfe"}
Oct 30 12:37:55.306328 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:55.306297    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf" event={"ID":"2050faf1-dfaf-4f24-9b62-112fe6957aed","Type":"ContainerStarted","Data":"7cf07a0272383fedb9ff34822e9898898063ec3da6c82104bbd69cc2e933a925"}
Oct 30 12:37:56.310994 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.310963    4429 generic.go:356] "Generic (PLEG): container finished" podID="2050faf1-dfaf-4f24-9b62-112fe6957aed" containerID="7cf07a0272383fedb9ff34822e9898898063ec3da6c82104bbd69cc2e933a925" exitCode=0
Oct 30 12:37:56.311402 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.311017    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf" event={"ID":"2050faf1-dfaf-4f24-9b62-112fe6957aed","Type":"ContainerDied","Data":"7cf07a0272383fedb9ff34822e9898898063ec3da6c82104bbd69cc2e933a925"}
Oct 30 12:37:56.584123 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.584100    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"]
Oct 30 12:37:56.590485 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.590466    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"
Oct 30 12:37:56.625354 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.625333    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:37:56.625844 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.625826    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-48897\""
Oct 30 12:37:56.626789 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.626772    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"]
Oct 30 12:37:56.641986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.641969    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 12:37:56.754158 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.754135    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhrj\" (UniqueName: \"kubernetes.io/projected/8a11a0a9-4c2c-43ce-a628-bbe665b03fec-kube-api-access-ddhrj\") pod \"jobset-operator-674cd59dc5-rvf99\" (UID: \"8a11a0a9-4c2c-43ce-a628-bbe665b03fec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"
Oct 30 12:37:56.754296 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.754167    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a11a0a9-4c2c-43ce-a628-bbe665b03fec-tmp\") pod \"jobset-operator-674cd59dc5-rvf99\" (UID: \"8a11a0a9-4c2c-43ce-a628-bbe665b03fec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"
Oct 30 12:37:56.855405 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.855344    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhrj\" (UniqueName: \"kubernetes.io/projected/8a11a0a9-4c2c-43ce-a628-bbe665b03fec-kube-api-access-ddhrj\") pod \"jobset-operator-674cd59dc5-rvf99\" (UID: \"8a11a0a9-4c2c-43ce-a628-bbe665b03fec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"
Oct 30 12:37:56.855405 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.855388    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a11a0a9-4c2c-43ce-a628-bbe665b03fec-tmp\") pod \"jobset-operator-674cd59dc5-rvf99\" (UID: \"8a11a0a9-4c2c-43ce-a628-bbe665b03fec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"
Oct 30 12:37:56.855713 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.855698    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a11a0a9-4c2c-43ce-a628-bbe665b03fec-tmp\") pod \"jobset-operator-674cd59dc5-rvf99\" (UID: \"8a11a0a9-4c2c-43ce-a628-bbe665b03fec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"
Oct 30 12:37:56.882429 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.882405    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhrj\" (UniqueName: \"kubernetes.io/projected/8a11a0a9-4c2c-43ce-a628-bbe665b03fec-kube-api-access-ddhrj\") pod \"jobset-operator-674cd59dc5-rvf99\" (UID: \"8a11a0a9-4c2c-43ce-a628-bbe665b03fec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"
Oct 30 12:37:56.931172 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:56.931150    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"
Oct 30 12:37:57.246928 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:57.246902    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99"]
Oct 30 12:37:57.247722 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:37:57.247693    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a11a0a9_4c2c_43ce_a628_bbe665b03fec.slice/crio-8d2b13bf598f61a3c0faa4dd5b7841ea4cc1418a94699011501c3a64227a3881 WatchSource:0}: Error finding container 8d2b13bf598f61a3c0faa4dd5b7841ea4cc1418a94699011501c3a64227a3881: Status 404 returned error can't find the container with id 8d2b13bf598f61a3c0faa4dd5b7841ea4cc1418a94699011501c3a64227a3881
Oct 30 12:37:57.316604 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:57.316573    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf" event={"ID":"2050faf1-dfaf-4f24-9b62-112fe6957aed","Type":"ContainerStarted","Data":"8702eb193f1d6ca175dea343aadff6ad9d423f777dd0ddbf5345754b3d8cb377"}
Oct 30 12:37:57.316963 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:57.316677    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:37:57.317903 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:57.317884    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99" event={"ID":"8a11a0a9-4c2c-43ce-a628-bbe665b03fec","Type":"ContainerStarted","Data":"8d2b13bf598f61a3c0faa4dd5b7841ea4cc1418a94699011501c3a64227a3881"}
Oct 30 12:37:57.391612 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:37:57.391565    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf" podStartSLOduration=2.965966454 podStartE2EDuration="6.391553738s" totalImagesPullingTime="2.964974165s" totalInitContainerRuntime="460.613119ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:57.385475991 +0000 UTC m=+335.440969815" watchObservedRunningTime="2026-10-30 12:37:57.391553738 +0000 UTC m=+335.447047562"
Oct 30 12:38:00.332002 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:00.331969    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99" event={"ID":"8a11a0a9-4c2c-43ce-a628-bbe665b03fec","Type":"ContainerStarted","Data":"b7bad6af1d935079edeed83d6ab7649003f35e552c2a19287aa0c1e5cf701264"}
Oct 30 12:38:00.386054 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:00.386004    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rvf99" podStartSLOduration=2.042524746 podStartE2EDuration="4.385990092s" totalImagesPullingTime="2.343465346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:00.385274128 +0000 UTC m=+338.440767955" watchObservedRunningTime="2026-10-30 12:38:00.385990092 +0000 UTC m=+338.441483915"
Oct 30 12:38:08.324934 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:08.324905    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c9d568b4f-tftjf"
Oct 30 12:38:12.369488 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:38:12.369450    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:38:13.744018 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.743990    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"]
Oct 30 12:38:13.754158 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.754118    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"
Oct 30 12:38:13.779844 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.779820    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-brg7b\""
Oct 30 12:38:13.789605 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.789575    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19761bbe-0bed-4ea5-bad0-c577e2b20480-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-q96sq\" (UID: \"19761bbe-0bed-4ea5-bad0-c577e2b20480\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"
Oct 30 12:38:13.789727 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.789620    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42gp\" (UniqueName: \"kubernetes.io/projected/19761bbe-0bed-4ea5-bad0-c577e2b20480-kube-api-access-m42gp\") pod \"cert-manager-cainjector-5c57bbd4fc-q96sq\" (UID: \"19761bbe-0bed-4ea5-bad0-c577e2b20480\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"
Oct 30 12:38:13.793800 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.793779    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 12:38:13.793887 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.793853    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 12:38:13.794885 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.794867    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"]
Oct 30 12:38:13.891167 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.891115    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19761bbe-0bed-4ea5-bad0-c577e2b20480-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-q96sq\" (UID: \"19761bbe-0bed-4ea5-bad0-c577e2b20480\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"
Oct 30 12:38:13.891334 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.891198    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42gp\" (UniqueName: \"kubernetes.io/projected/19761bbe-0bed-4ea5-bad0-c577e2b20480-kube-api-access-m42gp\") pod \"cert-manager-cainjector-5c57bbd4fc-q96sq\" (UID: \"19761bbe-0bed-4ea5-bad0-c577e2b20480\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"
Oct 30 12:38:13.960474 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.960447    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19761bbe-0bed-4ea5-bad0-c577e2b20480-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-q96sq\" (UID: \"19761bbe-0bed-4ea5-bad0-c577e2b20480\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"
Oct 30 12:38:13.974110 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:13.974078    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42gp\" (UniqueName: \"kubernetes.io/projected/19761bbe-0bed-4ea5-bad0-c577e2b20480-kube-api-access-m42gp\") pod \"cert-manager-cainjector-5c57bbd4fc-q96sq\" (UID: \"19761bbe-0bed-4ea5-bad0-c577e2b20480\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"
Oct 30 12:38:14.064953 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:14.064865    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"
Oct 30 12:38:14.383360 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:14.383332    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq"]
Oct 30 12:38:14.384872 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:38:14.384845    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19761bbe_0bed_4ea5_bad0_c577e2b20480.slice/crio-c9dc8656b68d6ec92a6f9a3779b6c4ce7c2b6673d259154b117db1ef286d1d97 WatchSource:0}: Error finding container c9dc8656b68d6ec92a6f9a3779b6c4ce7c2b6673d259154b117db1ef286d1d97: Status 404 returned error can't find the container with id c9dc8656b68d6ec92a6f9a3779b6c4ce7c2b6673d259154b117db1ef286d1d97
Oct 30 12:38:15.390839 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:15.390802    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq" event={"ID":"19761bbe-0bed-4ea5-bad0-c577e2b20480","Type":"ContainerStarted","Data":"c9dc8656b68d6ec92a6f9a3779b6c4ce7c2b6673d259154b117db1ef286d1d97"}
Oct 30 12:38:17.863447 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:17.863397    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"]
Oct 30 12:38:17.867855 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:17.867838    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:17.903877 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:17.903851    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-48fh9\""
Oct 30 12:38:17.904841 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:17.904822    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"]
Oct 30 12:38:17.921670 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:17.921648    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fbb3787-0ead-4228-bc7a-a8b85cddce02-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-549nm\" (UID: \"3fbb3787-0ead-4228-bc7a-a8b85cddce02\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:17.921785 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:17.921702    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvkl\" (UniqueName: \"kubernetes.io/projected/3fbb3787-0ead-4228-bc7a-a8b85cddce02-kube-api-access-dzvkl\") pod \"cert-manager-webhook-5dbbbf75cc-549nm\" (UID: \"3fbb3787-0ead-4228-bc7a-a8b85cddce02\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:18.022813 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:18.022787    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvkl\" (UniqueName: \"kubernetes.io/projected/3fbb3787-0ead-4228-bc7a-a8b85cddce02-kube-api-access-dzvkl\") pod \"cert-manager-webhook-5dbbbf75cc-549nm\" (UID: \"3fbb3787-0ead-4228-bc7a-a8b85cddce02\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:18.022903 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:18.022829    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fbb3787-0ead-4228-bc7a-a8b85cddce02-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-549nm\" (UID: \"3fbb3787-0ead-4228-bc7a-a8b85cddce02\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:18.053878 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:18.053851    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvkl\" (UniqueName: \"kubernetes.io/projected/3fbb3787-0ead-4228-bc7a-a8b85cddce02-kube-api-access-dzvkl\") pod \"cert-manager-webhook-5dbbbf75cc-549nm\" (UID: \"3fbb3787-0ead-4228-bc7a-a8b85cddce02\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:18.054211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:18.054193    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fbb3787-0ead-4228-bc7a-a8b85cddce02-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-549nm\" (UID: \"3fbb3787-0ead-4228-bc7a-a8b85cddce02\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:18.179591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:18.179529    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:18.491866 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:18.491845    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"]
Oct 30 12:38:18.494160 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:38:18.494130    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fbb3787_0ead_4228_bc7a_a8b85cddce02.slice/crio-ad660a8ff29c7c3338b153e473590df6d1e210d0599d3bd5f7299fe4987ef25d WatchSource:0}: Error finding container ad660a8ff29c7c3338b153e473590df6d1e210d0599d3bd5f7299fe4987ef25d: Status 404 returned error can't find the container with id ad660a8ff29c7c3338b153e473590df6d1e210d0599d3bd5f7299fe4987ef25d
Oct 30 12:38:19.420772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:19.420716    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm" event={"ID":"3fbb3787-0ead-4228-bc7a-a8b85cddce02","Type":"ContainerStarted","Data":"ad660a8ff29c7c3338b153e473590df6d1e210d0599d3bd5f7299fe4987ef25d"}
Oct 30 12:38:20.505072 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.505034    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-pprrm"]
Oct 30 12:38:20.510880 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.510852    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-pprrm"
Oct 30 12:38:20.526863 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.526808    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-pprrm"]
Oct 30 12:38:20.546242 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.546211    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23876c66-f1c7-463b-af85-4ff50af3838e-bound-sa-token\") pod \"cert-manager-597b95cb84-pprrm\" (UID: \"23876c66-f1c7-463b-af85-4ff50af3838e\") " pod="cert-manager/cert-manager-597b95cb84-pprrm"
Oct 30 12:38:20.546390 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.546277    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szhk\" (UniqueName: \"kubernetes.io/projected/23876c66-f1c7-463b-af85-4ff50af3838e-kube-api-access-4szhk\") pod \"cert-manager-597b95cb84-pprrm\" (UID: \"23876c66-f1c7-463b-af85-4ff50af3838e\") " pod="cert-manager/cert-manager-597b95cb84-pprrm"
Oct 30 12:38:20.548854 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.548822    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wh6d4\""
Oct 30 12:38:20.647621 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.647587    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szhk\" (UniqueName: \"kubernetes.io/projected/23876c66-f1c7-463b-af85-4ff50af3838e-kube-api-access-4szhk\") pod \"cert-manager-597b95cb84-pprrm\" (UID: \"23876c66-f1c7-463b-af85-4ff50af3838e\") " pod="cert-manager/cert-manager-597b95cb84-pprrm"
Oct 30 12:38:20.647787 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.647695    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23876c66-f1c7-463b-af85-4ff50af3838e-bound-sa-token\") pod \"cert-manager-597b95cb84-pprrm\" (UID: \"23876c66-f1c7-463b-af85-4ff50af3838e\") " pod="cert-manager/cert-manager-597b95cb84-pprrm"
Oct 30 12:38:20.690205 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.690175    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23876c66-f1c7-463b-af85-4ff50af3838e-bound-sa-token\") pod \"cert-manager-597b95cb84-pprrm\" (UID: \"23876c66-f1c7-463b-af85-4ff50af3838e\") " pod="cert-manager/cert-manager-597b95cb84-pprrm"
Oct 30 12:38:20.696830 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.696805    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szhk\" (UniqueName: \"kubernetes.io/projected/23876c66-f1c7-463b-af85-4ff50af3838e-kube-api-access-4szhk\") pod \"cert-manager-597b95cb84-pprrm\" (UID: \"23876c66-f1c7-463b-af85-4ff50af3838e\") " pod="cert-manager/cert-manager-597b95cb84-pprrm"
Oct 30 12:38:20.827221 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:20.827143    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-pprrm"
Oct 30 12:38:21.155648 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:21.155624    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-pprrm"]
Oct 30 12:38:21.158538 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:38:21.158504    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23876c66_f1c7_463b_af85_4ff50af3838e.slice/crio-3add78c9da5ab83ed86c2f35ff425f67758728c4c53e0c12c5008a327c21f116 WatchSource:0}: Error finding container 3add78c9da5ab83ed86c2f35ff425f67758728c4c53e0c12c5008a327c21f116: Status 404 returned error can't find the container with id 3add78c9da5ab83ed86c2f35ff425f67758728c4c53e0c12c5008a327c21f116
Oct 30 12:38:21.431009 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:21.430929    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-pprrm" event={"ID":"23876c66-f1c7-463b-af85-4ff50af3838e","Type":"ContainerStarted","Data":"3add78c9da5ab83ed86c2f35ff425f67758728c4c53e0c12c5008a327c21f116"}
Oct 30 12:38:22.436887 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:22.436788    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq" event={"ID":"19761bbe-0bed-4ea5-bad0-c577e2b20480","Type":"ContainerStarted","Data":"7f691881a3ffd3434ab8ede121df407d7831587d3196fc9c11fc9f84fa5d8a20"}
Oct 30 12:38:22.438632 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:22.438592    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-pprrm" event={"ID":"23876c66-f1c7-463b-af85-4ff50af3838e","Type":"ContainerStarted","Data":"09c153550bed95fec76f4e235bdab9c4f9f3c04fa3dc37f31d9956094befeca8"}
Oct 30 12:38:22.440226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:22.440199    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm" event={"ID":"3fbb3787-0ead-4228-bc7a-a8b85cddce02","Type":"ContainerStarted","Data":"7861fd4b8f2479178f6613784d6aa54f3ef9c3571bcda3fca21876c6e7f85012"}
Oct 30 12:38:22.440385 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:22.440251    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:22.511002 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:22.510939    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q96sq" podStartSLOduration=2.2423643699999998 podStartE2EDuration="9.510925129s" totalImagesPullingTime="7.268560759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:22.509896923 +0000 UTC m=+360.565390746" watchObservedRunningTime="2026-10-30 12:38:22.510925129 +0000 UTC m=+360.566418950"
Oct 30 12:38:22.557638 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:22.557581    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-pprrm" podStartSLOduration=2.029569179 podStartE2EDuration="2.557565974s" totalImagesPullingTime="527.996795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:22.555505947 +0000 UTC m=+360.610999771" watchObservedRunningTime="2026-10-30 12:38:22.557565974 +0000 UTC m=+360.613059795"
Oct 30 12:38:22.624202 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:22.624149    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm" podStartSLOduration=1.9348782469999999 podStartE2EDuration="5.624133623s" totalImagesPullingTime="3.689255376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:22.62375919 +0000 UTC m=+360.679253013" watchObservedRunningTime="2026-10-30 12:38:22.624133623 +0000 UTC m=+360.679627443"
Oct 30 12:38:25.129994 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.129968    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"]
Oct 30 12:38:25.133334 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.133311    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.165864 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.165840    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"]
Oct 30 12:38:25.166346 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.166329    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 12:38:25.166461 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.166389    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 12:38:25.182856 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.182831    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d39d81d-4605-4dec-8b57-bb683a61a741-tls-certs\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.182955 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.182885    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d39d81d-4605-4dec-8b57-bb683a61a741-tmp\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.183003 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.182967    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbjz\" (UniqueName: \"kubernetes.io/projected/2d39d81d-4605-4dec-8b57-bb683a61a741-kube-api-access-dbbjz\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.249247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.249223    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 12:38:25.249247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.249238    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 12:38:25.249247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.249225    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vnj47\""
Oct 30 12:38:25.283561 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.283539    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d39d81d-4605-4dec-8b57-bb683a61a741-tls-certs\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.283662 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.283576    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d39d81d-4605-4dec-8b57-bb683a61a741-tmp\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.283662 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.283608    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbjz\" (UniqueName: \"kubernetes.io/projected/2d39d81d-4605-4dec-8b57-bb683a61a741-kube-api-access-dbbjz\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.285871 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.285851    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d39d81d-4605-4dec-8b57-bb683a61a741-tmp\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.286167 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.286148    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d39d81d-4605-4dec-8b57-bb683a61a741-tls-certs\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.321743 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.321721    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbjz\" (UniqueName: \"kubernetes.io/projected/2d39d81d-4605-4dec-8b57-bb683a61a741-kube-api-access-dbbjz\") pod \"kube-auth-proxy-59688578c6-ds4tj\" (UID: \"2d39d81d-4605-4dec-8b57-bb683a61a741\") " pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.444795 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.444772    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"
Oct 30 12:38:25.743079 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:25.743046    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59688578c6-ds4tj"]
Oct 30 12:38:25.744129 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:38:25.744087    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d39d81d_4605_4dec_8b57_bb683a61a741.slice/crio-93c1b413c38bf6d370ab730228beb4a943982a4f4c4b7d5ea7204acaf6f2c677 WatchSource:0}: Error finding container 93c1b413c38bf6d370ab730228beb4a943982a4f4c4b7d5ea7204acaf6f2c677: Status 404 returned error can't find the container with id 93c1b413c38bf6d370ab730228beb4a943982a4f4c4b7d5ea7204acaf6f2c677
Oct 30 12:38:26.458684 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:26.458643    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj" event={"ID":"2d39d81d-4605-4dec-8b57-bb683a61a741","Type":"ContainerStarted","Data":"93c1b413c38bf6d370ab730228beb4a943982a4f4c4b7d5ea7204acaf6f2c677"}
Oct 30 12:38:28.447019 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:28.446992    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm"
Oct 30 12:38:28.468341 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:28.468302    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj" event={"ID":"2d39d81d-4605-4dec-8b57-bb683a61a741","Type":"ContainerStarted","Data":"15637ed6727825b29615a9967379d977a168115c2ccbef8cc5d18d8efa97fe56"}
Oct 30 12:38:28.541264 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:28.541224    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59688578c6-ds4tj" podStartSLOduration=1.3261912 podStartE2EDuration="3.541211908s" totalImagesPullingTime="2.215020708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:28.540403544 +0000 UTC m=+366.595897369" watchObservedRunningTime="2026-10-30 12:38:28.541211908 +0000 UTC m=+366.596705732"
Oct 30 12:38:42.371486 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:38:42.371451    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:38:55.542533 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.542492    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"]
Oct 30 12:38:55.550532 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.550509    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.563818 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.563790    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"]
Oct 30 12:38:55.576536 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.576513    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 12:38:55.576644 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.576599    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 12:38:55.580786 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.580767    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 12:38:55.580893 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.580774    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-6t4vd\""
Oct 30 12:38:55.724528 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.724501    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a781be-ab69-465c-9f95-e06ab080ec84-metrics-certs\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.724646 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.724534    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10a781be-ab69-465c-9f95-e06ab080ec84-cert\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.724646 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.724551    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk2zh\" (UniqueName: \"kubernetes.io/projected/10a781be-ab69-465c-9f95-e06ab080ec84-kube-api-access-gk2zh\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.724720 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.724661    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/10a781be-ab69-465c-9f95-e06ab080ec84-manager-config\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.825913 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.825849    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/10a781be-ab69-465c-9f95-e06ab080ec84-manager-config\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.825913 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.825901    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a781be-ab69-465c-9f95-e06ab080ec84-metrics-certs\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.826072 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.825925    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10a781be-ab69-465c-9f95-e06ab080ec84-cert\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.826072 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.825941    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk2zh\" (UniqueName: \"kubernetes.io/projected/10a781be-ab69-465c-9f95-e06ab080ec84-kube-api-access-gk2zh\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.826484 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.826465    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/10a781be-ab69-465c-9f95-e06ab080ec84-manager-config\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.828708 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.828685    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10a781be-ab69-465c-9f95-e06ab080ec84-cert\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.828894 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.828876    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a781be-ab69-465c-9f95-e06ab080ec84-metrics-certs\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.850991 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.850948    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk2zh\" (UniqueName: \"kubernetes.io/projected/10a781be-ab69-465c-9f95-e06ab080ec84-kube-api-access-gk2zh\") pod \"jobset-controller-manager-69cd49d574-qj754\" (UID: \"10a781be-ab69-465c-9f95-e06ab080ec84\") " pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:55.864458 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:55.864437    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:56.197646 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:56.197624    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"]
Oct 30 12:38:56.198961 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:38:56.198940    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a781be_ab69_465c_9f95_e06ab080ec84.slice/crio-e666a866cdcff1efa67ed975ed63d9ffaf42d08f7b7562ac9bb39ff0d11071cb WatchSource:0}: Error finding container e666a866cdcff1efa67ed975ed63d9ffaf42d08f7b7562ac9bb39ff0d11071cb: Status 404 returned error can't find the container with id e666a866cdcff1efa67ed975ed63d9ffaf42d08f7b7562ac9bb39ff0d11071cb
Oct 30 12:38:56.590366 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:56.590304    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754" event={"ID":"10a781be-ab69-465c-9f95-e06ab080ec84","Type":"ContainerStarted","Data":"e666a866cdcff1efa67ed975ed63d9ffaf42d08f7b7562ac9bb39ff0d11071cb"}
Oct 30 12:38:59.605683 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:59.605643    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754" event={"ID":"10a781be-ab69-465c-9f95-e06ab080ec84","Type":"ContainerStarted","Data":"9393dd3b7ebb2fca48962bf3e5cf6e465cb98a5914ecfc2b042bd10ed36ba33a"}
Oct 30 12:38:59.606105 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:59.605796    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:38:59.650007 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:38:59.649948    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754" podStartSLOduration=1.753340243 podStartE2EDuration="4.649934908s" totalImagesPullingTime="2.896594665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:38:59.649756935 +0000 UTC m=+397.705250760" watchObservedRunningTime="2026-10-30 12:38:59.649934908 +0000 UTC m=+397.705428732"
Oct 30 12:39:10.615368 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:39:10.615265    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-69cd49d574-qj754"
Oct 30 12:39:12.371506 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:39:12.371456    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:39:42.370970 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:39:42.370933    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:40:12.370716 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:40:12.370681    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:40:42.369995 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:40:42.369913    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:41:12.370008 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:41:12.369974    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:41:42.370268 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:41:42.370234    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:42:12.370347 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:42:12.370243    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:42:42.369954 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:42:42.369919    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:43:12.370994 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:43:12.370958    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:43:42.370666 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:43:42.370585    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:44:12.370523 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:44:12.370490    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:44:42.370220 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:44:42.370183    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:45:12.370597 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:45:12.370506    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:45:42.370600 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:45:42.370565    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:46:12.370213 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:46:12.370166    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:46:42.370308 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:46:42.370214    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:47:12.370098 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:47:12.370064    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:47:42.370140 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:47:42.370105    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:48:12.371131 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:48:12.371060    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:48:42.372000 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:48:42.371966    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:49:12.372499 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:49:12.372469    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:49:42.373023 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:49:42.372935    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:49:45.942849 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.942817    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ddfbb87d4-bkzx2"]
Oct 30 12:49:45.946189 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.946167    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:45.968489 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.968470    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 12:49:45.979915 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.979891    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 12:49:45.980059 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.980011    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 12:49:45.980207 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.980189    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-75gjs\""
Oct 30 12:49:45.980461 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.980443    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 12:49:45.982264 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.982247    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 12:49:45.983607 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:45.983588    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ddfbb87d4-bkzx2"]
Oct 30 12:49:46.076943 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.076925    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45e3a6bf-a04a-4001-8036-28397efa666c-webhook-cert\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.077054 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.076993    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e3a6bf-a04a-4001-8036-28397efa666c-metrics-certs\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.077128 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.077108    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxj66\" (UniqueName: \"kubernetes.io/projected/45e3a6bf-a04a-4001-8036-28397efa666c-kube-api-access-lxj66\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.178215 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.178192    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e3a6bf-a04a-4001-8036-28397efa666c-metrics-certs\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.178329 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.178226    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxj66\" (UniqueName: \"kubernetes.io/projected/45e3a6bf-a04a-4001-8036-28397efa666c-kube-api-access-lxj66\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.178329 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.178250    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45e3a6bf-a04a-4001-8036-28397efa666c-webhook-cert\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.180677 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.180658    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45e3a6bf-a04a-4001-8036-28397efa666c-webhook-cert\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.180763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.180710    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45e3a6bf-a04a-4001-8036-28397efa666c-metrics-certs\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.203146 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.203088    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxj66\" (UniqueName: \"kubernetes.io/projected/45e3a6bf-a04a-4001-8036-28397efa666c-kube-api-access-lxj66\") pod \"maas-controller-7ddfbb87d4-bkzx2\" (UID: \"45e3a6bf-a04a-4001-8036-28397efa666c\") " pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.259078 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.259038    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:46.564856 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.564833    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ddfbb87d4-bkzx2"]
Oct 30 12:49:46.566258 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:49:46.566232    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e3a6bf_a04a_4001_8036_28397efa666c.slice/crio-3866d24da952294edf85f1d648e446f1ba1ea678cb9af354b8305dd78fafb591 WatchSource:0}: Error finding container 3866d24da952294edf85f1d648e446f1ba1ea678cb9af354b8305dd78fafb591: Status 404 returned error can't find the container with id 3866d24da952294edf85f1d648e446f1ba1ea678cb9af354b8305dd78fafb591
Oct 30 12:49:46.567454 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:46.567439    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:49:47.192820 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:47.192780    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2" event={"ID":"45e3a6bf-a04a-4001-8036-28397efa666c","Type":"ContainerStarted","Data":"3866d24da952294edf85f1d648e446f1ba1ea678cb9af354b8305dd78fafb591"}
Oct 30 12:49:49.202787 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:49.202754    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2" event={"ID":"45e3a6bf-a04a-4001-8036-28397efa666c","Type":"ContainerStarted","Data":"4127d9970982e4905f3b2ae8576c7b885e6d86774283c710adedf446b0612aa0"}
Oct 30 12:49:49.203258 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:49.202870    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:49:50.728278 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.728224    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2" podStartSLOduration=3.436906054 podStartE2EDuration="5.728209863s" totalImagesPullingTime="2.291303809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:49.25191056 +0000 UTC m=+1047.307404383" watchObservedRunningTime="2026-10-30 12:49:50.728209863 +0000 UTC m=+1048.783703688"
Oct 30 12:49:50.729319 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.729277    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"]
Oct 30 12:49:50.732601 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.732587    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.762910 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.762761    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 12:49:50.762910 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.762783    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 12:49:50.763098 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.762914    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-sfw2p\""
Oct 30 12:49:50.765955 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.765932    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"]
Oct 30 12:49:50.818606 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.818575    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c383851d-6fc2-4523-9a52-b603e4149172-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.818752 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.818614    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c383851d-6fc2-4523-9a52-b603e4149172-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.818752 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.818645    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2nm\" (UniqueName: \"kubernetes.io/projected/c383851d-6fc2-4523-9a52-b603e4149172-kube-api-access-lf2nm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.818855 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.818783    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c383851d-6fc2-4523-9a52-b603e4149172-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.853477 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.853457    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 12:49:50.920018 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.919992    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c383851d-6fc2-4523-9a52-b603e4149172-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.920119 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.920039    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c383851d-6fc2-4523-9a52-b603e4149172-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.920119 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.920095    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2nm\" (UniqueName: \"kubernetes.io/projected/c383851d-6fc2-4523-9a52-b603e4149172-kube-api-access-lf2nm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.920233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.920159    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c383851d-6fc2-4523-9a52-b603e4149172-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.920758 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.920737    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c383851d-6fc2-4523-9a52-b603e4149172-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.920967 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.920947    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c383851d-6fc2-4523-9a52-b603e4149172-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.922668 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.922651    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c383851d-6fc2-4523-9a52-b603e4149172-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:50.944560 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:50.944539    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2nm\" (UniqueName: \"kubernetes.io/projected/c383851d-6fc2-4523-9a52-b603e4149172-kube-api-access-lf2nm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rcx5n\" (UID: \"c383851d-6fc2-4523-9a52-b603e4149172\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:51.044715 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:51.044659    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:49:51.343766 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:51.343746    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"]
Oct 30 12:49:51.345593 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:49:51.345562    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc383851d_6fc2_4523_9a52_b603e4149172.slice/crio-4e93a93082e70f4314e9a16885bb96f24d36f3ab904c7b5d7524489771bedd07 WatchSource:0}: Error finding container 4e93a93082e70f4314e9a16885bb96f24d36f3ab904c7b5d7524489771bedd07: Status 404 returned error can't find the container with id 4e93a93082e70f4314e9a16885bb96f24d36f3ab904c7b5d7524489771bedd07
Oct 30 12:49:52.215352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:49:52.215320    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n" event={"ID":"c383851d-6fc2-4523-9a52-b603e4149172","Type":"ContainerStarted","Data":"4e93a93082e70f4314e9a16885bb96f24d36f3ab904c7b5d7524489771bedd07"}
Oct 30 12:50:00.209076 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:00.209049    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ddfbb87d4-bkzx2"
Oct 30 12:50:03.269447 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:03.269418    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n" event={"ID":"c383851d-6fc2-4523-9a52-b603e4149172","Type":"ContainerStarted","Data":"12bbfe34332cf6f0f33153af6a3cf746f9c324e11dc37d9dca11a12e83a77d9c"}
Oct 30 12:50:03.269855 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:03.269534    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:50:03.314910 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:03.314861    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n" podStartSLOduration=1.812744684 podStartE2EDuration="13.314848566s" totalImagesPullingTime="11.502103882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:03.314042873 +0000 UTC m=+1061.369536697" watchObservedRunningTime="2026-10-30 12:50:03.314848566 +0000 UTC m=+1061.370342388"
Oct 30 12:50:12.370485 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:50:12.370448    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:50:14.279481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:14.279454    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rcx5n"
Oct 30 12:50:30.235353 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.235316    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"]
Oct 30 12:50:30.674456 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.674377    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"]
Oct 30 12:50:30.674627 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.674497    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.693197 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.693173    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:50:30.693599 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.693583    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:50:30.693684 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.693606    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dlqhq\""
Oct 30 12:50:30.693684 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.693666    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 12:50:30.693684 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.693676    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 12:50:30.730431 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.730394    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f55165-93cb-4197-935e-a6f2e6cbc44a-metrics-certs\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.730431 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.730437    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f55165-93cb-4197-935e-a6f2e6cbc44a-maas-api-tls\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.730683 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.730598    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f55165-93cb-4197-935e-a6f2e6cbc44a-openshift-service-ca\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.730683 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.730642    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2rb\" (UniqueName: \"kubernetes.io/projected/f1f55165-93cb-4197-935e-a6f2e6cbc44a-kube-api-access-8n2rb\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.831653 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.831625    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f55165-93cb-4197-935e-a6f2e6cbc44a-maas-api-tls\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.831850 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.831681    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f55165-93cb-4197-935e-a6f2e6cbc44a-openshift-service-ca\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.831850 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.831712    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2rb\" (UniqueName: \"kubernetes.io/projected/f1f55165-93cb-4197-935e-a6f2e6cbc44a-kube-api-access-8n2rb\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.831850 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.831743    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f55165-93cb-4197-935e-a6f2e6cbc44a-metrics-certs\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.832546 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.832477    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f55165-93cb-4197-935e-a6f2e6cbc44a-openshift-service-ca\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.834239 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.834215    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1f55165-93cb-4197-935e-a6f2e6cbc44a-metrics-certs\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.834398 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.834381    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1f55165-93cb-4197-935e-a6f2e6cbc44a-maas-api-tls\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:30.857007 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:30.856986    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2rb\" (UniqueName: \"kubernetes.io/projected/f1f55165-93cb-4197-935e-a6f2e6cbc44a-kube-api-access-8n2rb\") pod \"maas-api-78747dffff-qqr9g\" (UID: \"f1f55165-93cb-4197-935e-a6f2e6cbc44a\") " pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:31.008387 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:31.008351    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:31.322015 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:31.321993    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"]
Oct 30 12:50:31.323378 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:50:31.323353    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f55165_93cb_4197_935e_a6f2e6cbc44a.slice/crio-4e5a706e881a94cd83121c319896e893f2c892096f16427ad711951cc78e55c7 WatchSource:0}: Error finding container 4e5a706e881a94cd83121c319896e893f2c892096f16427ad711951cc78e55c7: Status 404 returned error can't find the container with id 4e5a706e881a94cd83121c319896e893f2c892096f16427ad711951cc78e55c7
Oct 30 12:50:31.383469 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:31.383445    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g" event={"ID":"f1f55165-93cb-4197-935e-a6f2e6cbc44a","Type":"ContainerStarted","Data":"4e5a706e881a94cd83121c319896e893f2c892096f16427ad711951cc78e55c7"}
Oct 30 12:50:33.393533 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:33.393491    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g" event={"ID":"f1f55165-93cb-4197-935e-a6f2e6cbc44a","Type":"ContainerStarted","Data":"b1c21ac645f9dc1eb932f826dff0f3c54728f59d11c1b5ed2b2c6f7d37717551"}
Oct 30 12:50:33.393933 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:33.393580    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:33.440126 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:33.440077    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g" podStartSLOduration=2.071184858 podStartE2EDuration="3.440065522s" totalImagesPullingTime="1.368880664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:33.439883571 +0000 UTC m=+1091.495377377" watchObservedRunningTime="2026-10-30 12:50:33.440065522 +0000 UTC m=+1091.495559391"
Oct 30 12:50:35.504771 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.504738    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"]
Oct 30 12:50:35.508346 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.508323    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.523136 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.523113    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"]
Oct 30 12:50:35.526278 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.526258    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.535566 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.535547    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tcq9j\""
Oct 30 12:50:35.535566 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.535548    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 12:50:35.540572 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.540552    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"]
Oct 30 12:50:35.553458 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.553436    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"]
Oct 30 12:50:35.571812 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.571792    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.571937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.571821    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.571937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.571846    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.571937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.571869    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.572060 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572011    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.572060 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572040    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.572158 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572070    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.572158 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572103    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.572262 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572203    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.572262 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572247    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.572392 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572314    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.572392 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572331    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dbl\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-kube-api-access-k9dbl\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.572392 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572379    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.572548 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572425    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.572548 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572456    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.572548 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572484    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zr6\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-kube-api-access-t7zr6\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.572548 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572506    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616efda-cc2c-4127-885e-49ea7d47fc12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.572670 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.572560    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673099    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673133    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673152    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673250    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dbl\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-kube-api-access-k9dbl\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673319    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673357    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673384    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673410    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zr6\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-kube-api-access-t7zr6\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673434    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616efda-cc2c-4127-885e-49ea7d47fc12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673475    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673535    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673562    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673595    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673642    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673715    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673785    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673687    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.673972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.673830    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674002    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674015    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674010    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674247    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616efda-cc2c-4127-885e-49ea7d47fc12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674334    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674461    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674761    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674874    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.674996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.674924    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.675249 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.675114    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.676232 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.676209    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.676366 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.676338    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.676523 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.676507    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.676759 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.676741    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.701343 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.701314    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dbl\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-kube-api-access-k9dbl\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.701465 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.701389    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zr6\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-kube-api-access-t7zr6\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.701465 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.701431    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-hxkhh\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.701681 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.701655    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-sjm4f\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:35.821604 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.821549    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:35.837604 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:35.837583    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:36.144947 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:36.144918    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"]
Oct 30 12:50:36.145405 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:50:36.145375    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3f5772_16c2_4650_8bd8_fdb0ec149974.slice/crio-27be810bfab56f482e3e269bef2cd91644af7d565e3aa940c9b65b733fd76780 WatchSource:0}: Error finding container 27be810bfab56f482e3e269bef2cd91644af7d565e3aa940c9b65b733fd76780: Status 404 returned error can't find the container with id 27be810bfab56f482e3e269bef2cd91644af7d565e3aa940c9b65b733fd76780
Oct 30 12:50:36.147392 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:36.147367    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"]
Oct 30 12:50:36.155468 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:50:36.155444    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3616efda_cc2c_4127_885e_49ea7d47fc12.slice/crio-b49b3029f00e931baa77572acfa8ae9039116354dcd133e9ba2c5bc024ba7494 WatchSource:0}: Error finding container b49b3029f00e931baa77572acfa8ae9039116354dcd133e9ba2c5bc024ba7494: Status 404 returned error can't find the container with id b49b3029f00e931baa77572acfa8ae9039116354dcd133e9ba2c5bc024ba7494
Oct 30 12:50:36.407737 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:36.407656    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" event={"ID":"9a3f5772-16c2-4650-8bd8-fdb0ec149974","Type":"ContainerStarted","Data":"27be810bfab56f482e3e269bef2cd91644af7d565e3aa940c9b65b733fd76780"}
Oct 30 12:50:36.408826 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:36.408807    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" event={"ID":"3616efda-cc2c-4127-885e-49ea7d47fc12","Type":"ContainerStarted","Data":"b49b3029f00e931baa77572acfa8ae9039116354dcd133e9ba2c5bc024ba7494"}
Oct 30 12:50:39.044502 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.044466    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:39.044790 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.044554    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:39.044790 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.044597    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:39.053978 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.053953    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:39.054076 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.054024    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:39.054076 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.054054    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:50:39.403313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.403218    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78747dffff-qqr9g"
Oct 30 12:50:39.457450 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.457384    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" event={"ID":"9a3f5772-16c2-4650-8bd8-fdb0ec149974","Type":"ContainerStarted","Data":"e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f"}
Oct 30 12:50:39.514478 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.514435    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" event={"ID":"3616efda-cc2c-4127-885e-49ea7d47fc12","Type":"ContainerStarted","Data":"40fffc686568d0dd8264e6c80807decd64715be0fe332256d692f81ad731d1db"}
Oct 30 12:50:39.579390 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.566519    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podStartSLOduration=1.661105473 podStartE2EDuration="4.566497011s" totalImagesPullingTime="2.905391538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:39.515062271 +0000 UTC m=+1097.570556098" watchObservedRunningTime="2026-10-30 12:50:39.566497011 +0000 UTC m=+1097.621990836"
Oct 30 12:50:39.580652 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.580596    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podStartSLOduration=1.694418251 podStartE2EDuration="4.580579951s" totalImagesPullingTime="2.8861617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:39.580270993 +0000 UTC m=+1097.635764809" watchObservedRunningTime="2026-10-30 12:50:39.580579951 +0000 UTC m=+1097.636073776"
Oct 30 12:50:39.822448 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.822419    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:39.838596 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:39.838547    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:40.823877 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:40.823837    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-hxkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 12:50:40.824299 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:40.823908    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 12:50:40.839203 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:40.839179    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-sjm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 12:50:40.839342 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:40.839221    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 12:50:41.822085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:41.822047    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-hxkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 12:50:41.822238 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:41.822102    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 12:50:41.838918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:41.838888    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-sjm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 12:50:41.839274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:41.838943    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 12:50:42.418455 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:50:42.418422    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:50:42.822889 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:42.822854    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-hxkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 12:50:42.823070 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:42.822917    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 12:50:42.838376 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:42.838351    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-sjm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 12:50:42.838505 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:42.838426    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 12:50:43.822736 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:43.822701    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-hxkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 12:50:43.823113 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:43.822761    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 12:50:43.838726 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:43.838700    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-sjm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 12:50:43.838846 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:43.838735    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 12:50:44.821861 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:44.821831    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-hxkhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 12:50:44.822119 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:44.822089    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 12:50:44.838277 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:44.838253    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-sjm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 12:50:44.838605 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:44.838316    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 12:50:45.826301 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:45.826253    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:45.826765 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:45.826748    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:45.827497 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:45.827476    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 12:50:45.842008 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:45.841990    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:45.842339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:45.842240    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:50:45.842937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:50:45.842920    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 12:51:12.380039 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:51:12.379936    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:51:42.371145 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:51:42.371105    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:51:54.148731 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.148694    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"]
Oct 30 12:51:54.152357 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.152334    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.169680 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.169644    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 12:51:54.183618 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.183597    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2vklm\""
Oct 30 12:51:54.183618 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.183550    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 12:51:54.184108 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.184093    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 12:51:54.186589 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.186570    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"]
Oct 30 12:51:54.188482 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.188464    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.188530 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.188497    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wkb\" (UniqueName: \"kubernetes.io/projected/037e455b-6b2a-4cf5-a8e8-0d606071d12c-kube-api-access-s9wkb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.188586 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.188571    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.188620 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.188603    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037e455b-6b2a-4cf5-a8e8-0d606071d12c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.188652 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.188621    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.188758 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.188739    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.289556 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.289526    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.289701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.289563    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037e455b-6b2a-4cf5-a8e8-0d606071d12c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.289701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.289587    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.289701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.289645    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.289861 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.289713    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.289861 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.289747    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wkb\" (UniqueName: \"kubernetes.io/projected/037e455b-6b2a-4cf5-a8e8-0d606071d12c-kube-api-access-s9wkb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.290039 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.290019    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.290089 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.290069    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.290129 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.290089    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.291902 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.291886    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037e455b-6b2a-4cf5-a8e8-0d606071d12c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.292144 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.292127    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037e455b-6b2a-4cf5-a8e8-0d606071d12c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.316006 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.315984    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wkb\" (UniqueName: \"kubernetes.io/projected/037e455b-6b2a-4cf5-a8e8-0d606071d12c-kube-api-access-s9wkb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9d754\" (UID: \"037e455b-6b2a-4cf5-a8e8-0d606071d12c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.466583 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.466551    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:54.763660 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.763628    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"]
Oct 30 12:51:54.764199 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:51:54.764174    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037e455b_6b2a_4cf5_a8e8_0d606071d12c.slice/crio-d8478619935be83a5574dab5a3576fb25d5ae4a89f3c7ae9d8a34648de04001a WatchSource:0}: Error finding container d8478619935be83a5574dab5a3576fb25d5ae4a89f3c7ae9d8a34648de04001a: Status 404 returned error can't find the container with id d8478619935be83a5574dab5a3576fb25d5ae4a89f3c7ae9d8a34648de04001a
Oct 30 12:51:54.838832 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:54.838807    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754" event={"ID":"037e455b-6b2a-4cf5-a8e8-0d606071d12c","Type":"ContainerStarted","Data":"d8478619935be83a5574dab5a3576fb25d5ae4a89f3c7ae9d8a34648de04001a"}
Oct 30 12:51:56.850054 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.850015    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754" event={"ID":"037e455b-6b2a-4cf5-a8e8-0d606071d12c","Type":"ContainerStarted","Data":"0a9a4445ed4a6383e6a05ac43f80b37c292071c48840fe301ca0eb83eb3391a0"}
Oct 30 12:51:56.894072 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.894040    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"]
Oct 30 12:51:56.898041 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.898020    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:56.912696 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.912656    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754" podStartSLOduration=1.5266512030000001 podStartE2EDuration="2.912645201s" totalImagesPullingTime="1.385993998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:56.912279304 +0000 UTC m=+1174.967773125" watchObservedRunningTime="2026-10-30 12:51:56.912645201 +0000 UTC m=+1174.968139025"
Oct 30 12:51:56.914686 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.914661    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msml\" (UniqueName: \"kubernetes.io/projected/cd0b29e6-33d5-4327-836f-57c1eea4e635-kube-api-access-9msml\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:56.914870 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.914849    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:56.914942 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.914882    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:56.915003 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.914945    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:56.915056 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.915035    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0b29e6-33d5-4327-836f-57c1eea4e635-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:56.915098 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.915093    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:56.927066 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.927049    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 12:51:56.928748 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:56.928727    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"]
Oct 30 12:51:57.015527 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.015498    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msml\" (UniqueName: \"kubernetes.io/projected/cd0b29e6-33d5-4327-836f-57c1eea4e635-kube-api-access-9msml\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.015660 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.015541    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.015660 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.015568    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.015660 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.015602    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.015660 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.015642    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0b29e6-33d5-4327-836f-57c1eea4e635-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.015878 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.015700    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.015989 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.015966    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.016054 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.015987    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.016184 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.016166    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.018093 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.018066    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0b29e6-33d5-4327-836f-57c1eea4e635-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.018349 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.018333    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0b29e6-33d5-4327-836f-57c1eea4e635-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.040322 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.040300    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msml\" (UniqueName: \"kubernetes.io/projected/cd0b29e6-33d5-4327-836f-57c1eea4e635-kube-api-access-9msml\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-gjcgq\" (UID: \"cd0b29e6-33d5-4327-836f-57c1eea4e635\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.210885 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.210857    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:51:57.467379 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.467315    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:51:57.506924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.506899    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"]
Oct 30 12:51:57.507607 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:51:57.507582    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0b29e6_33d5_4327_836f_57c1eea4e635.slice/crio-a8891091fc3b944d3ecff157a0a88f519ffef7de4be5cf7bd67c647fcbc2f9e4 WatchSource:0}: Error finding container a8891091fc3b944d3ecff157a0a88f519ffef7de4be5cf7bd67c647fcbc2f9e4: Status 404 returned error can't find the container with id a8891091fc3b944d3ecff157a0a88f519ffef7de4be5cf7bd67c647fcbc2f9e4
Oct 30 12:51:57.856380 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.856334    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq" event={"ID":"cd0b29e6-33d5-4327-836f-57c1eea4e635","Type":"ContainerStarted","Data":"3477092e362178d62dd78be4c97f88d40ad2b65bc5caaac937cc7e7d0ea206cd"}
Oct 30 12:51:57.856753 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.856399    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq" event={"ID":"cd0b29e6-33d5-4327-836f-57c1eea4e635","Type":"ContainerStarted","Data":"a8891091fc3b944d3ecff157a0a88f519ffef7de4be5cf7bd67c647fcbc2f9e4"}
Oct 30 12:51:57.905778 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:57.905732    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq" podStartSLOduration=1.7320834189999998 podStartE2EDuration="1.905717609s" totalImagesPullingTime="173.63419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:57.904429469 +0000 UTC m=+1175.959923292" watchObservedRunningTime="2026-10-30 12:51:57.905717609 +0000 UTC m=+1175.961211434"
Oct 30 12:51:58.210996 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:51:58.210969    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:52:00.291448 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.291409    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c6qkj"]
Oct 30 12:52:00.295138 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.295114    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.311591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.311569    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 12:52:00.326975 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.326951    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c6qkj"]
Oct 30 12:52:00.341672 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.341638    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.341772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.341684    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-home\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.341772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.341716    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.341876 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.341786    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.341945 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.341923    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jdwf\" (UniqueName: \"kubernetes.io/projected/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-kube-api-access-6jdwf\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.342038 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.342020    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443092 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443066    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443210 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443100    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443210 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443133    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-home\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443210 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443167    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443210 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443181    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443423 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443209    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jdwf\" (UniqueName: \"kubernetes.io/projected/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-kube-api-access-6jdwf\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443615    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-home\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443713 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443688    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.443771 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.443730    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.445655 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.445633    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.445791 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.445775    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.470571 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.470549    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jdwf\" (UniqueName: \"kubernetes.io/projected/1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e-kube-api-access-6jdwf\") pod \"echo-simulator-kserve-64b6f8f96c-c6qkj\" (UID: \"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.609363 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.609304    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:00.910699 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:00.910663    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c6qkj"]
Oct 30 12:52:00.911640 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:52:00.911616    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faa86f9_abf0_4dd5_ab7e_c087ffe1ea7e.slice/crio-a7cf3699c1cd5f923f43a6f09018b23cdf856f0a7348e1efd130d4a89eaf47b2 WatchSource:0}: Error finding container a7cf3699c1cd5f923f43a6f09018b23cdf856f0a7348e1efd130d4a89eaf47b2: Status 404 returned error can't find the container with id a7cf3699c1cd5f923f43a6f09018b23cdf856f0a7348e1efd130d4a89eaf47b2
Oct 30 12:52:01.922741 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:01.922695    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj" event={"ID":"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e","Type":"ContainerStarted","Data":"a7cf3699c1cd5f923f43a6f09018b23cdf856f0a7348e1efd130d4a89eaf47b2"}
Oct 30 12:52:02.929477 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:02.929444    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj" event={"ID":"1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e","Type":"ContainerStarted","Data":"a93baa3d3bed3c73ba76e4689915f57136f8f423baa4b5eb1017e414cb73e7c0"}
Oct 30 12:52:02.982531 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:02.982470    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj" podStartSLOduration=1.070666715 podStartE2EDuration="2.982453074s" totalImagesPullingTime="1.911786359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:02.980930208 +0000 UTC m=+1181.036424031" watchObservedRunningTime="2026-10-30 12:52:02.982453074 +0000 UTC m=+1181.037946910"
Oct 30 12:52:03.609537 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:03.609505    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:04.467350 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:04.467315    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:52:04.479688 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:04.479665    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:52:04.951448 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:04.951421    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9d754"
Oct 30 12:52:05.760092 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.760065    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"]
Oct 30 12:52:05.763899 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.763882    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.788919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.788889    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"]
Oct 30 12:52:05.813651 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.813621    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.814204 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.814184    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.814372 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.814222    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.814372 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.814303    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.814372 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.814355    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.814541 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.814422    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.814541 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.814457    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.814541 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.814483    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.814705 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.814535    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxtw\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-kube-api-access-msxtw\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.915711 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.915679    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.915853 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.915771    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.915853 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.915807    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.915853 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.915833    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916025 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.915889    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxtw\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-kube-api-access-msxtw\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916025 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.915927    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916025 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.915953    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916025 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.915984    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916025 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.916013    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916262 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.916077    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916262 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.916159    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916262 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.916248    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916507 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.916484    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.916667 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.916630    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.918331 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.918306    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.918565 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.918548    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.949733 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.949707    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:05.949959 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:05.949938    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxtw\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-kube-api-access-msxtw\") pod \"maas-default-gateway-openshift-default-57f78bcd55-fks52\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:06.077067 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:06.076993    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:06.400309 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:06.400254    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"]
Oct 30 12:52:06.400418 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:52:06.400306    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5229a88_b9ba_40ea_b08c_91dc4f1bc1fb.slice/crio-24816d24d3d6b8edd626b21ba33e8e93041ca6eca924fb43cc0a0f7e51bb8dae WatchSource:0}: Error finding container 24816d24d3d6b8edd626b21ba33e8e93041ca6eca924fb43cc0a0f7e51bb8dae: Status 404 returned error can't find the container with id 24816d24d3d6b8edd626b21ba33e8e93041ca6eca924fb43cc0a0f7e51bb8dae
Oct 30 12:52:06.402776 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:06.402731    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:06.402870 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:06.402818    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:06.402919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:06.402865    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:06.955188 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:06.955149    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" event={"ID":"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb","Type":"ContainerStarted","Data":"82e8d7aa3835a93308f4b670e847daca6053137d5a7c342ebfb4ed1b1d0d4c91"}
Oct 30 12:52:06.955812 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:06.955786    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" event={"ID":"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb","Type":"ContainerStarted","Data":"24816d24d3d6b8edd626b21ba33e8e93041ca6eca924fb43cc0a0f7e51bb8dae"}
Oct 30 12:52:07.033893 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:07.032867    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podStartSLOduration=2.032847257 podStartE2EDuration="2.032847257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:07.031902615 +0000 UTC m=+1185.087396430" watchObservedRunningTime="2026-10-30 12:52:07.032847257 +0000 UTC m=+1185.088341274"
Oct 30 12:52:07.081194 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:07.081157    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:07.081550 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:07.081525    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 12:52:07.081753 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:07.081722    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 12:52:07.214684 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:07.214598    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:52:07.227857 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:07.227834    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:52:08.012105 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:08.012071    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-gjcgq"
Oct 30 12:52:08.078638 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:08.078604    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 12:52:08.078877 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:08.078850    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 12:52:09.077679 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:09.077630    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 12:52:09.078146 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:09.077684    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 12:52:10.077969 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:10.077938    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 12:52:10.078348 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:10.077993    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 12:52:10.609695 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:10.609653    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:10.622421 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:10.622398    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:11.003428 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:11.003401    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-c6qkj"
Oct 30 12:52:11.077656 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:11.077617    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 12:52:11.077817 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:11.077688    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 12:52:12.077386 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:12.077355    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 12:52:12.077860 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:12.077410    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 12:52:12.391252 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:52:12.391172    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:52:13.077867 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:13.077831    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 12:52:13.078267 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:13.077890    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 12:52:14.696619 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:14.696582    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:14.697434 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:14.697411    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:14.698214 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:14.698194    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 12:52:20.813799 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.813769    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"]
Oct 30 12:52:20.819859 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.819837    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.855514 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.855488    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"]
Oct 30 12:52:20.871162 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871138    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4qc\" (UniqueName: \"kubernetes.io/projected/c513e96c-4db5-43e5-bbd1-eda409197969-kube-api-access-zk4qc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.871336 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871175    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c513e96c-4db5-43e5-bbd1-eda409197969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.871336 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871208    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.871336 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871320    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.871503 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871375    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c513e96c-4db5-43e5-bbd1-eda409197969-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.871503 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871470    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.871572 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871533    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c513e96c-4db5-43e5-bbd1-eda409197969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.871572 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871558    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.871656 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.871597    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.972655 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.972631    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c513e96c-4db5-43e5-bbd1-eda409197969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.972760 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.972662    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.972760 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.972697    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.972840 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.972802    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4qc\" (UniqueName: \"kubernetes.io/projected/c513e96c-4db5-43e5-bbd1-eda409197969-kube-api-access-zk4qc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.972888 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.972853    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c513e96c-4db5-43e5-bbd1-eda409197969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.972944 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.972908    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.972944 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.972935    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.973035 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.972958    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c513e96c-4db5-43e5-bbd1-eda409197969-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.973035 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.973021    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.973138 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.973058    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.973265 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.973242    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.973392 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.973375    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.973455 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.973436    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.973501 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.973460    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c513e96c-4db5-43e5-bbd1-eda409197969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.975766 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.975743    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c513e96c-4db5-43e5-bbd1-eda409197969-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.976063 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.976042    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c513e96c-4db5-43e5-bbd1-eda409197969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.997798 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.997777    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4qc\" (UniqueName: \"kubernetes.io/projected/c513e96c-4db5-43e5-bbd1-eda409197969-kube-api-access-zk4qc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:20.997907 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:20.997885    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c513e96c-4db5-43e5-bbd1-eda409197969-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qvm42\" (UID: \"c513e96c-4db5-43e5-bbd1-eda409197969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:21.134194 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:21.134139    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:21.454424 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:52:21.454398    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc513e96c_4db5_43e5_bbd1_eda409197969.slice/crio-8d3fd7eb96cda571c424ef9b012f5792b8d2b7dc32be735d6228314d31072309 WatchSource:0}: Error finding container 8d3fd7eb96cda571c424ef9b012f5792b8d2b7dc32be735d6228314d31072309: Status 404 returned error can't find the container with id 8d3fd7eb96cda571c424ef9b012f5792b8d2b7dc32be735d6228314d31072309
Oct 30 12:52:21.456337 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:21.456311    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"]
Oct 30 12:52:21.457121 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:21.457094    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:21.457172 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:21.457153    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:21.457207 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:21.457180    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:22.044079 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:22.044045    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" event={"ID":"c513e96c-4db5-43e5-bbd1-eda409197969","Type":"ContainerStarted","Data":"0e9706218238d90fdbb3fececb3ddb220a788d7f03caa2fb8a434a3b730041ed"}
Oct 30 12:52:22.044531 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:22.044137    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" event={"ID":"c513e96c-4db5-43e5-bbd1-eda409197969","Type":"ContainerStarted","Data":"8d3fd7eb96cda571c424ef9b012f5792b8d2b7dc32be735d6228314d31072309"}
Oct 30 12:52:22.123988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:22.123934    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podStartSLOduration=2.123916438 podStartE2EDuration="2.123916438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:22.122102011 +0000 UTC m=+1200.177595835" watchObservedRunningTime="2026-10-30 12:52:22.123916438 +0000 UTC m=+1200.179410263"
Oct 30 12:52:22.135075 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:22.135053    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:22.136490 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:22.136465    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qvm42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 12:52:22.136602 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:22.136517    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podUID="c513e96c-4db5-43e5-bbd1-eda409197969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 12:52:23.134982 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:23.134947    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qvm42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 12:52:23.135376 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:23.134997    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podUID="c513e96c-4db5-43e5-bbd1-eda409197969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 12:52:24.135435 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:24.135401    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qvm42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 12:52:24.135832 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:24.135450    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podUID="c513e96c-4db5-43e5-bbd1-eda409197969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 12:52:25.134588 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:25.134557    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qvm42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 12:52:25.134761 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:25.134611    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podUID="c513e96c-4db5-43e5-bbd1-eda409197969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 12:52:26.135217 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:26.135186    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qvm42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 12:52:26.135611 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:26.135235    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podUID="c513e96c-4db5-43e5-bbd1-eda409197969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 12:52:27.135459 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:27.135426    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qvm42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 12:52:27.135828 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:27.135474    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podUID="c513e96c-4db5-43e5-bbd1-eda409197969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 12:52:28.135299 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:28.135257    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qvm42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 12:52:28.135471 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:28.135342    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podUID="c513e96c-4db5-43e5-bbd1-eda409197969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 12:52:29.658169 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:29.658142    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:29.658675 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:29.658604    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:29.659361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:29.659342    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42"
Oct 30 12:52:42.370211 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:52:42.370110    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:52:55.504629 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.504597    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"]
Oct 30 12:52:55.508556 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.508536    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.526950 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.526930    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f1c4045f-openshift-default-dockercfg-rgn5j\""
Oct 30 12:52:55.542379 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.542349    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"]
Oct 30 12:52:55.687863 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.687834    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-certs\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.688010 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.687894    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-token\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.688068 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.688014    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrff\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-kube-api-access-nkrff\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.688129 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.688071    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-podinfo\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.688172 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.688161    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-envoy\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.688217 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.688189    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istiod-ca-cert\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.688254 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.688225    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-socket\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.688400 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.688381    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-data\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.688467 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.688449    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-credential-socket\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.789807 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.789726    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-data\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.789807 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.789764    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-credential-socket\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.789807 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.789799    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-certs\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790094 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.789861    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-token\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790094 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.789897    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrff\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-kube-api-access-nkrff\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790094 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.789937    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-podinfo\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790094 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.789971    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-envoy\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790094 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.789996    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istiod-ca-cert\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790094 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.790029    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-socket\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790427 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.790161    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-data\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790427 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.790180    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-certs\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790427 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.790229    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-credential-socket\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790427 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.790401    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-socket\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.790742 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.790724    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istiod-ca-cert\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.792442 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.792415    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-envoy\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.792651 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.792635    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-podinfo\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.816265 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.816245    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-token\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.819599 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.819576    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrff\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-kube-api-access-nkrff\") pod \"e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") " pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:55.822245 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:55.822226    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:56.172904 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:56.172877    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"]
Oct 30 12:52:56.173094 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:52:56.173073    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4fe4bf_7fa0_4c45_8472_4d1ca74b9e68.slice/crio-7a59991e5846f18ab305ccbe3382fb08cca46ecd0208841cc69b26bf13a91f29 WatchSource:0}: Error finding container 7a59991e5846f18ab305ccbe3382fb08cca46ecd0208841cc69b26bf13a91f29: Status 404 returned error can't find the container with id 7a59991e5846f18ab305ccbe3382fb08cca46ecd0208841cc69b26bf13a91f29
Oct 30 12:52:56.175272 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:56.175242    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:56.175394 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:56.175321    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:56.175394 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:56.175356    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:56.212946 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:56.212917    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" event={"ID":"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68","Type":"ContainerStarted","Data":"7a59991e5846f18ab305ccbe3382fb08cca46ecd0208841cc69b26bf13a91f29"}
Oct 30 12:52:57.219731 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.219693    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" event={"ID":"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68","Type":"ContainerStarted","Data":"0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d"}
Oct 30 12:52:57.284264 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.284220    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podStartSLOduration=2.284206485 podStartE2EDuration="2.284206485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:57.28275151 +0000 UTC m=+1235.338245353" watchObservedRunningTime="2026-10-30 12:52:57.284206485 +0000 UTC m=+1235.339700309"
Oct 30 12:52:57.385392 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.385359    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"]
Oct 30 12:52:57.389526 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.389505    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.403862 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.403836    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc6de86-242b-4566-8d86-3e190fa55b61-istiod-ca-cert\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.403958 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.403905    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-envoy\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.404007 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.403967    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47d2\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-kube-api-access-k47d2\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.404045 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.404012    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc6de86-242b-4566-8d86-3e190fa55b61-istio-podinfo\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.404086 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.404052    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-socket\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.404131 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.404117    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-istio-token\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.404166 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.404141    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-credential-socket\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.404212 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.404172    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-data\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.404254 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.404233    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-certs\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.410335 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.410122    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-873e4f54-openshift-default-dockercfg-28zqr\""
Oct 30 12:52:57.413211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.413190    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"]
Oct 30 12:52:57.505198 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505106    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc6de86-242b-4566-8d86-3e190fa55b61-istio-podinfo\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.505198 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505176    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-socket\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.505443 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505222    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-istio-token\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.505443 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505253    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-credential-socket\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.505443 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505306    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-data\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.505443 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505334    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-certs\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.505443 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505371    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc6de86-242b-4566-8d86-3e190fa55b61-istiod-ca-cert\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.505443 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505434    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-envoy\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.505785 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505476    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47d2\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-kube-api-access-k47d2\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.506014 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.505977    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-data\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.506132 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.506080    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-certs\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.506258 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.506237    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-socket\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.506704 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.506682    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-credential-socket\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.506805 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.506758    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc6de86-242b-4566-8d86-3e190fa55b61-istiod-ca-cert\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.509097 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.509073    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc6de86-242b-4566-8d86-3e190fa55b61-istio-podinfo\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.509178 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.509132    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-envoy\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.533934 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.533906    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-istio-token\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.534543 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.534519    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47d2\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-kube-api-access-k47d2\") pod \"e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") " pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.704135 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.704103    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:57.822960 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.822931    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:57.827691 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:57.827671    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:58.016725 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.016702    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"]
Oct 30 12:52:58.017874 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:52:58.017853    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc6de86_242b_4566_8d86_3e190fa55b61.slice/crio-9f0e6faffdaef1f762b3715d28c5667ce1635e9351d2da4d076f768ecc119d21 WatchSource:0}: Error finding container 9f0e6faffdaef1f762b3715d28c5667ce1635e9351d2da4d076f768ecc119d21: Status 404 returned error can't find the container with id 9f0e6faffdaef1f762b3715d28c5667ce1635e9351d2da4d076f768ecc119d21
Oct 30 12:52:58.020359 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.020330    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:58.020440 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.020403    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:58.020440 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.020430    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:52:58.225633 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.225596    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" event={"ID":"edc6de86-242b-4566-8d86-3e190fa55b61","Type":"ContainerStarted","Data":"d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1"}
Oct 30 12:52:58.226101 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.225654    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" event={"ID":"edc6de86-242b-4566-8d86-3e190fa55b61","Type":"ContainerStarted","Data":"9f0e6faffdaef1f762b3715d28c5667ce1635e9351d2da4d076f768ecc119d21"}
Oct 30 12:52:58.226101 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.225941    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:58.226668 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.226648    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:52:58.275494 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.275410    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" podStartSLOduration=1.275397816 podStartE2EDuration="1.275397816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:58.27387104 +0000 UTC m=+1236.329364864" watchObservedRunningTime="2026-10-30 12:52:58.275397816 +0000 UTC m=+1236.330891640"
Oct 30 12:52:58.705172 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:58.705144    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:52:59.709492 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:52:59.709464    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:00.235247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.235214    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:00.236252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.236236    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:00.758864 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.758820    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"]
Oct 30 12:53:00.768447 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.768420    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:00.783022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.782996    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"]
Oct 30 12:53:00.787643 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.787621    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fb8b5ba5\""
Oct 30 12:53:00.837695 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.837658    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babf1051-8ea7-4e6b-86b7-c1233548a4c1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s\" (UID: \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:00.837871 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.837757    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bb9\" (UniqueName: \"kubernetes.io/projected/babf1051-8ea7-4e6b-86b7-c1233548a4c1-kube-api-access-q6bb9\") pod \"payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s\" (UID: \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:00.853375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.853348    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fb8b5ba5-dockercfg-nmx7b\""
Oct 30 12:53:00.938428 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.938398    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babf1051-8ea7-4e6b-86b7-c1233548a4c1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s\" (UID: \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:00.938583 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.938454    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bb9\" (UniqueName: \"kubernetes.io/projected/babf1051-8ea7-4e6b-86b7-c1233548a4c1-kube-api-access-q6bb9\") pod \"payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s\" (UID: \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:00.939077 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.939057    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babf1051-8ea7-4e6b-86b7-c1233548a4c1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s\" (UID: \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:00.963649 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:00.963625    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bb9\" (UniqueName: \"kubernetes.io/projected/babf1051-8ea7-4e6b-86b7-c1233548a4c1-kube-api-access-q6bb9\") pod \"payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s\" (UID: \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\") " pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:01.082569 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:01.082484    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:01.384218 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:01.384190    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"]
Oct 30 12:53:01.384799 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:01.384780    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabf1051_8ea7_4e6b_86b7_c1233548a4c1.slice/crio-07edfbba9b85ee933e12ebf4501fafb069463989a0798d314b1e1e73dc4cafb7 WatchSource:0}: Error finding container 07edfbba9b85ee933e12ebf4501fafb069463989a0798d314b1e1e73dc4cafb7: Status 404 returned error can't find the container with id 07edfbba9b85ee933e12ebf4501fafb069463989a0798d314b1e1e73dc4cafb7
Oct 30 12:53:02.173156 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.173110    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"]
Oct 30 12:53:02.177343 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.177320    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.203527 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.201558    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f1c4045f-serving-cert\""
Oct 30 12:53:02.205021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.204674    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"]
Oct 30 12:53:02.255978 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.255914    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-maas-api-tls\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.256365 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.256345    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-metrics-certs\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.256956 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.256935    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5275fd1a-2e74-4f2b-936c-5036f155b424-openshift-service-ca\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.257157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.257100    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkffr\" (UniqueName: \"kubernetes.io/projected/5275fd1a-2e74-4f2b-936c-5036f155b424-kube-api-access-xkffr\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.259725 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.259693    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s" event={"ID":"babf1051-8ea7-4e6b-86b7-c1233548a4c1","Type":"ContainerStarted","Data":"07edfbba9b85ee933e12ebf4501fafb069463989a0798d314b1e1e73dc4cafb7"}
Oct 30 12:53:02.358836 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.358797    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-maas-api-tls\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.359000 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.358849    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-metrics-certs\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.359115 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.359091    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5275fd1a-2e74-4f2b-936c-5036f155b424-openshift-service-ca\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.359165 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.359136    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkffr\" (UniqueName: \"kubernetes.io/projected/5275fd1a-2e74-4f2b-936c-5036f155b424-kube-api-access-xkffr\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.359774 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.359747    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5275fd1a-2e74-4f2b-936c-5036f155b424-openshift-service-ca\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.363877 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.363851    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-maas-api-tls\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.365577 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.365555    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-metrics-certs\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.384345 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.384323    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkffr\" (UniqueName: \"kubernetes.io/projected/5275fd1a-2e74-4f2b-936c-5036f155b424-kube-api-access-xkffr\") pod \"maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.507060 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.507030    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:02.898125 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:02.898097    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"]
Oct 30 12:53:02.898972 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:02.898946    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5275fd1a_2e74_4f2b_936c_5036f155b424.slice/crio-5e192e55e47b4dd9b61e1a1d95be98ec1fd33f02a3fa3abc9767913885494e95 WatchSource:0}: Error finding container 5e192e55e47b4dd9b61e1a1d95be98ec1fd33f02a3fa3abc9767913885494e95: Status 404 returned error can't find the container with id 5e192e55e47b4dd9b61e1a1d95be98ec1fd33f02a3fa3abc9767913885494e95
Oct 30 12:53:03.267368 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:03.267332    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9" event={"ID":"5275fd1a-2e74-4f2b-936c-5036f155b424","Type":"ContainerStarted","Data":"5e192e55e47b4dd9b61e1a1d95be98ec1fd33f02a3fa3abc9767913885494e95"}
Oct 30 12:53:04.274400 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:04.274352    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s" event={"ID":"babf1051-8ea7-4e6b-86b7-c1233548a4c1","Type":"ContainerStarted","Data":"83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91"}
Oct 30 12:53:04.274823 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:04.274539    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:04.276233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:04.276208    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9" event={"ID":"5275fd1a-2e74-4f2b-936c-5036f155b424","Type":"ContainerStarted","Data":"865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6"}
Oct 30 12:53:04.276434 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:04.276418    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:04.317959 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:04.317919    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s" podStartSLOduration=2.248022227 podStartE2EDuration="4.317907104s" totalImagesPullingTime="2.069884877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:04.317820851 +0000 UTC m=+1242.373314676" watchObservedRunningTime="2026-10-30 12:53:04.317907104 +0000 UTC m=+1242.373400946"
Oct 30 12:53:04.375393 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:04.375354    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9" podStartSLOduration=1.85360006 podStartE2EDuration="2.375341063s" totalImagesPullingTime="521.741003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:04.374421197 +0000 UTC m=+1242.429915020" watchObservedRunningTime="2026-10-30 12:53:04.375341063 +0000 UTC m=+1242.430834905"
Oct 30 12:53:05.851191 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.851157    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"]
Oct 30 12:53:05.855319 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.855272    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.887353 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.887322    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-873e4f54-serving-cert\""
Oct 30 12:53:05.887825 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.887801    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"]
Oct 30 12:53:05.896278 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.894524    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-maas-api-tls\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.896278 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.894637    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-metrics-certs\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.896278 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.894734    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jpn\" (UniqueName: \"kubernetes.io/projected/415fb8df-7084-4bf4-9dd9-7920637accad-kube-api-access-77jpn\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.896278 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.894770    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fb8df-7084-4bf4-9dd9-7920637accad-openshift-service-ca\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.995936 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.995895    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-maas-api-tls\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.996111 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.995977    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-metrics-certs\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.996111 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.996031    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jpn\" (UniqueName: \"kubernetes.io/projected/415fb8df-7084-4bf4-9dd9-7920637accad-kube-api-access-77jpn\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.996111 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.996067    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fb8df-7084-4bf4-9dd9-7920637accad-openshift-service-ca\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.996965 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.996935    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fb8df-7084-4bf4-9dd9-7920637accad-openshift-service-ca\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:05.999587 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:05.999562    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-maas-api-tls\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:06.000120 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:06.000003    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-metrics-certs\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:06.022341 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:06.022316    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jpn\" (UniqueName: \"kubernetes.io/projected/415fb8df-7084-4bf4-9dd9-7920637accad-kube-api-access-77jpn\") pod \"maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:06.168030 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:06.167955    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:06.500487 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:06.500465    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"]
Oct 30 12:53:06.501773 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:06.501742    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415fb8df_7084_4bf4_9dd9_7920637accad.slice/crio-6cfcaec778fd5e4a1da6df6e15e64fbbee9375850be07dcafe37d8ebad68803c WatchSource:0}: Error finding container 6cfcaec778fd5e4a1da6df6e15e64fbbee9375850be07dcafe37d8ebad68803c: Status 404 returned error can't find the container with id 6cfcaec778fd5e4a1da6df6e15e64fbbee9375850be07dcafe37d8ebad68803c
Oct 30 12:53:07.293324 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:07.293264    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2" event={"ID":"415fb8df-7084-4bf4-9dd9-7920637accad","Type":"ContainerStarted","Data":"607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad"}
Oct 30 12:53:07.293701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:07.293361    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:07.293701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:07.293388    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2" event={"ID":"415fb8df-7084-4bf4-9dd9-7920637accad","Type":"ContainerStarted","Data":"6cfcaec778fd5e4a1da6df6e15e64fbbee9375850be07dcafe37d8ebad68803c"}
Oct 30 12:53:07.339085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:07.339034    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2" podStartSLOduration=1.9699385569999999 podStartE2EDuration="2.339017685s" totalImagesPullingTime="369.079128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:07.337690167 +0000 UTC m=+1245.393183973" watchObservedRunningTime="2026-10-30 12:53:07.339017685 +0000 UTC m=+1245.394511536"
Oct 30 12:53:09.677396 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.677355    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"]
Oct 30 12:53:09.681655 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.681634    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:09.699151 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.699114    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-98f58556\""
Oct 30 12:53:09.704059 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.704034    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"]
Oct 30 12:53:09.713013 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.712992    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-98f58556-dockercfg-vhg88\""
Oct 30 12:53:09.744073 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.744037    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c729fc62-18ca-4d3a-9887-b7a77537603d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm\" (UID: \"c729fc62-18ca-4d3a-9887-b7a77537603d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:09.744190 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.744124    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgh7\" (UniqueName: \"kubernetes.io/projected/c729fc62-18ca-4d3a-9887-b7a77537603d-kube-api-access-2vgh7\") pod \"payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm\" (UID: \"c729fc62-18ca-4d3a-9887-b7a77537603d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:09.845753 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.845717    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c729fc62-18ca-4d3a-9887-b7a77537603d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm\" (UID: \"c729fc62-18ca-4d3a-9887-b7a77537603d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:09.845952 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.845775    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgh7\" (UniqueName: \"kubernetes.io/projected/c729fc62-18ca-4d3a-9887-b7a77537603d-kube-api-access-2vgh7\") pod \"payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm\" (UID: \"c729fc62-18ca-4d3a-9887-b7a77537603d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:09.846436 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.846416    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c729fc62-18ca-4d3a-9887-b7a77537603d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm\" (UID: \"c729fc62-18ca-4d3a-9887-b7a77537603d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:09.875202 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.875176    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgh7\" (UniqueName: \"kubernetes.io/projected/c729fc62-18ca-4d3a-9887-b7a77537603d-kube-api-access-2vgh7\") pod \"payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm\" (UID: \"c729fc62-18ca-4d3a-9887-b7a77537603d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:09.998158 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:09.998125    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:10.287167 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:10.287094    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:10.331752 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:10.331328    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"]
Oct 30 12:53:10.334476 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:10.334439    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc729fc62_18ca_4d3a_9887_b7a77537603d.slice/crio-37e8c545d30f1bab041ff8634ce33e2d9b65511db8250c20b72f50cceeacde3b WatchSource:0}: Error finding container 37e8c545d30f1bab041ff8634ce33e2d9b65511db8250c20b72f50cceeacde3b: Status 404 returned error can't find the container with id 37e8c545d30f1bab041ff8634ce33e2d9b65511db8250c20b72f50cceeacde3b
Oct 30 12:53:11.313744 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:11.313716    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm" event={"ID":"c729fc62-18ca-4d3a-9887-b7a77537603d","Type":"ContainerStarted","Data":"e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb"}
Oct 30 12:53:11.314190 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:11.314172    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm" event={"ID":"c729fc62-18ca-4d3a-9887-b7a77537603d","Type":"ContainerStarted","Data":"37e8c545d30f1bab041ff8634ce33e2d9b65511db8250c20b72f50cceeacde3b"}
Oct 30 12:53:11.315049 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:11.315033    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:12.416764 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:12.416734    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:53:13.304874 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:13.304836    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:13.358311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:13.356603    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm" podStartSLOduration=4.356584627 podStartE2EDuration="4.356584627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:11.365219783 +0000 UTC m=+1249.420713608" watchObservedRunningTime="2026-10-30 12:53:13.356584627 +0000 UTC m=+1251.412078451"
Oct 30 12:53:14.272746 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:14.272412    4429 patch_prober.go:29] interesting pod/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:53:14.272746 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:14.272484    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:53:15.285846 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.285815    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:15.290704 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.290681    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"]
Oct 30 12:53:15.294470 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.294450    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.320675 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.320654    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-98f58556-serving-cert\""
Oct 30 12:53:15.321775 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.321746    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"]
Oct 30 12:53:15.442278 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.442240    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-metrics-certs\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.442458 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.442369    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9mr\" (UniqueName: \"kubernetes.io/projected/f7bff32f-7499-41f2-8348-66bcfa97f0dc-kube-api-access-lx9mr\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.442458 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.442436    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bff32f-7499-41f2-8348-66bcfa97f0dc-openshift-service-ca\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.442536 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.442481    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-maas-api-tls\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.543408 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.543318    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-metrics-certs\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.543408 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.543392    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9mr\" (UniqueName: \"kubernetes.io/projected/f7bff32f-7499-41f2-8348-66bcfa97f0dc-kube-api-access-lx9mr\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.543602 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.543425    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bff32f-7499-41f2-8348-66bcfa97f0dc-openshift-service-ca\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.543602 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.543455    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-maas-api-tls\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.544226 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.544177    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bff32f-7499-41f2-8348-66bcfa97f0dc-openshift-service-ca\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.546275 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.546252    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-metrics-certs\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.546624 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.546606    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-maas-api-tls\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.588184 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.587192    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9mr\" (UniqueName: \"kubernetes.io/projected/f7bff32f-7499-41f2-8348-66bcfa97f0dc-kube-api-access-lx9mr\") pod \"maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.608912 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.608877    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:15.933193 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:15.933161    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"]
Oct 30 12:53:15.933901 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:15.933867    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bff32f_7499_41f2_8348_66bcfa97f0dc.slice/crio-2d35e9accc13b68a5d184b5afd6f723da296a1eee6e13160cdb3465a67f1ffc6 WatchSource:0}: Error finding container 2d35e9accc13b68a5d184b5afd6f723da296a1eee6e13160cdb3465a67f1ffc6: Status 404 returned error can't find the container with id 2d35e9accc13b68a5d184b5afd6f723da296a1eee6e13160cdb3465a67f1ffc6
Oct 30 12:53:16.380198 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:16.380166    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d" event={"ID":"f7bff32f-7499-41f2-8348-66bcfa97f0dc","Type":"ContainerStarted","Data":"2d35e9accc13b68a5d184b5afd6f723da296a1eee6e13160cdb3465a67f1ffc6"}
Oct 30 12:53:17.388051 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.388015    4429 generic.go:356] "Generic (PLEG): container finished" podID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerID="0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d" exitCode=0
Oct 30 12:53:17.388606 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.388083    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" event={"ID":"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68","Type":"ContainerDied","Data":"0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d"}
Oct 30 12:53:17.388606 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.388544    4429 scope.go:122] "RemoveContainer" containerID="0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d"
Oct 30 12:53:17.389865 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.389831    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:17.389965 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.389925    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:17.390022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.389973    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:17.390099 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.389971    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d" event={"ID":"f7bff32f-7499-41f2-8348-66bcfa97f0dc","Type":"ContainerStarted","Data":"4ae12d4ede84f9c0c34aa07255d39e17a211376955b23d8230cc60963a3e4682"}
Oct 30 12:53:17.390215 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.390126    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:17.502277 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:17.502219    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d" podStartSLOduration=2.018922841 podStartE2EDuration="2.502200884s" totalImagesPullingTime="483.278043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:17.499712177 +0000 UTC m=+1255.555206002" watchObservedRunningTime="2026-10-30 12:53:17.502200884 +0000 UTC m=+1255.557694710"
Oct 30 12:53:18.396461 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:18.396424    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" event={"ID":"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68","Type":"ContainerStarted","Data":"5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c"}
Oct 30 12:53:18.823346 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:18.823305    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:53:18.823633 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:18.823608    4429 patch_prober.go:29] interesting pod/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:53:18.823693 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:18.823671    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 12:53:19.823116 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:19.823079    4429 patch_prober.go:29] interesting pod/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:53:19.823519 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:19.823133    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 12:53:20.823266 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:20.823191    4429 patch_prober.go:29] interesting pod/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:53:20.823266 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:20.823243    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 12:53:21.416339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:21.416304    4429 generic.go:356] "Generic (PLEG): container finished" podID="edc6de86-242b-4566-8d86-3e190fa55b61" containerID="d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1" exitCode=0
Oct 30 12:53:21.416535 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:21.416395    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" event={"ID":"edc6de86-242b-4566-8d86-3e190fa55b61","Type":"ContainerDied","Data":"d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1"}
Oct 30 12:53:21.416865 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:21.416849    4429 scope.go:122] "RemoveContainer" containerID="d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1"
Oct 30 12:53:21.418172 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:21.418146    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:21.418232 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:21.418216    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:21.418269 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:21.418246    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:21.823100 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:21.823063    4429 patch_prober.go:29] interesting pod/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:53:21.823266 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:21.823131    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 12:53:22.423414 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:22.423385    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" event={"ID":"edc6de86-242b-4566-8d86-3e190fa55b61","Type":"ContainerStarted","Data":"d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b"}
Oct 30 12:53:22.708779 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:22.708746    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:22.709116 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:22.709096    4429 patch_prober.go:29] interesting pod/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 12:53:22.709256 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:22.709235    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 12:53:22.822650 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:22.822621    4429 patch_prober.go:29] interesting pod/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:53:22.822867 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:22.822849    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 12:53:23.327872 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:23.327846    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:23.405804 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:23.405771    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:23.704647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:23.704610    4429 patch_prober.go:29] interesting pod/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 12:53:23.705137 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:23.704659    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 12:53:23.827622 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:23.827592    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:53:23.827897 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:23.827880    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:53:23.828646 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:23.828625    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:53:24.705405 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:24.705373    4429 patch_prober.go:29] interesting pod/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 12:53:24.705794 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:24.705425    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 12:53:25.705118 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:25.705081    4429 patch_prober.go:29] interesting pod/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 12:53:25.705353 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:25.705141    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 12:53:26.705395 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:26.705358    4429 patch_prober.go:29] interesting pod/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 12:53:26.705893 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:26.705414    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 12:53:27.709221 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:27.709188    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:27.709736 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:27.709623    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:27.710421 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:27.710403    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:29.638080 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:29.638046    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"]
Oct 30 12:53:32.612971 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.612930    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"]
Oct 30 12:53:32.613490 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.613182    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9" podUID="5275fd1a-2e74-4f2b-936c-5036f155b424" containerName="maas-api" containerID="cri-o://865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6" gracePeriod=30
Oct 30 12:53:32.881821 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.881415    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:32.983913 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.983632    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-metrics-certs\") pod \"5275fd1a-2e74-4f2b-936c-5036f155b424\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") "
Oct 30 12:53:32.983913 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.983755    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5275fd1a-2e74-4f2b-936c-5036f155b424-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5275fd1a-2e74-4f2b-936c-5036f155b424-openshift-service-ca\") pod \"5275fd1a-2e74-4f2b-936c-5036f155b424\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") "
Oct 30 12:53:32.983913 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.983791    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5275fd1a-2e74-4f2b-936c-5036f155b424-kube-api-access-xkffr\" (UniqueName: \"kubernetes.io/projected/5275fd1a-2e74-4f2b-936c-5036f155b424-kube-api-access-xkffr\") pod \"5275fd1a-2e74-4f2b-936c-5036f155b424\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") "
Oct 30 12:53:32.983913 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.983822    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-maas-api-tls\") pod \"5275fd1a-2e74-4f2b-936c-5036f155b424\" (UID: \"5275fd1a-2e74-4f2b-936c-5036f155b424\") "
Oct 30 12:53:32.984345 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.984165    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5275fd1a-2e74-4f2b-936c-5036f155b424-openshift-service-ca" pod "5275fd1a-2e74-4f2b-936c-5036f155b424" (UID: "5275fd1a-2e74-4f2b-936c-5036f155b424"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:32.984345 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.984310    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5275fd1a-2e74-4f2b-936c-5036f155b424-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:32.986424 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.986384    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5275fd1a-2e74-4f2b-936c-5036f155b424-kube-api-access-xkffr" pod "5275fd1a-2e74-4f2b-936c-5036f155b424" (UID: "5275fd1a-2e74-4f2b-936c-5036f155b424"). InnerVolumeSpecName "kube-api-access-xkffr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:32.986424 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.986399    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-metrics-certs" pod "5275fd1a-2e74-4f2b-936c-5036f155b424" (UID: "5275fd1a-2e74-4f2b-936c-5036f155b424"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:32.986585 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:32.986445    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-maas-api-tls" pod "5275fd1a-2e74-4f2b-936c-5036f155b424" (UID: "5275fd1a-2e74-4f2b-936c-5036f155b424"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:33.085186 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.085153    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkffr\" (UniqueName: \"kubernetes.io/projected/5275fd1a-2e74-4f2b-936c-5036f155b424-kube-api-access-xkffr\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:33.085186 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.085180    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:33.085186 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.085191    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5275fd1a-2e74-4f2b-936c-5036f155b424-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:33.484786 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.484756    4429 generic.go:356] "Generic (PLEG): container finished" podID="5275fd1a-2e74-4f2b-936c-5036f155b424" containerID="865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6" exitCode=0
Oct 30 12:53:33.484965 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.484807    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"
Oct 30 12:53:33.484965 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.484829    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9" event={"ID":"5275fd1a-2e74-4f2b-936c-5036f155b424","Type":"ContainerDied","Data":"865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6"}
Oct 30 12:53:33.484965 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.484874    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9" event={"ID":"5275fd1a-2e74-4f2b-936c-5036f155b424","Type":"ContainerDied","Data":"5e192e55e47b4dd9b61e1a1d95be98ec1fd33f02a3fa3abc9767913885494e95"}
Oct 30 12:53:33.484965 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.484899    4429 scope.go:122] "RemoveContainer" containerID="865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6"
Oct 30 12:53:33.495077 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.495060    4429 scope.go:122] "RemoveContainer" containerID="865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6"
Oct 30 12:53:33.495383 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:33.495362    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6\": container with ID starting with 865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6 not found: ID does not exist" containerID="865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6"
Oct 30 12:53:33.495437 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.495392    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6"} err="failed to get container status \"865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6\": rpc error: code = NotFound desc = could not find container \"865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6\": container with ID starting with 865ea1e8cea4570adccc214755aa29078782abaf91dbe1efb491d618b3048ec6 not found: ID does not exist"
Oct 30 12:53:33.540811 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.540786    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"]
Oct 30 12:53:33.554593 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:33.554568    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f1c4045f-85bf944797-cmhr9"]
Oct 30 12:53:34.075117 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:34.075085    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5275fd1a-2e74-4f2b-936c-5036f155b424" path="/var/lib/kubelet/pods/5275fd1a-2e74-4f2b-936c-5036f155b424/volumes"
Oct 30 12:53:34.638773 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:34.638731    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s" podUID="babf1051-8ea7-4e6b-86b7-c1233548a4c1" containerName="payload-processing" containerID="cri-o://83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91" gracePeriod=25
Oct 30 12:53:34.887984 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:34.887954    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:35.003646 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.003621    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babf1051-8ea7-4e6b-86b7-c1233548a4c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babf1051-8ea7-4e6b-86b7-c1233548a4c1-plugins-config-volume\") pod \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\" (UID: \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\") "
Oct 30 12:53:35.003834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.003781    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babf1051-8ea7-4e6b-86b7-c1233548a4c1-kube-api-access-q6bb9\" (UniqueName: \"kubernetes.io/projected/babf1051-8ea7-4e6b-86b7-c1233548a4c1-kube-api-access-q6bb9\") pod \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\" (UID: \"babf1051-8ea7-4e6b-86b7-c1233548a4c1\") "
Oct 30 12:53:35.004514 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.004481    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babf1051-8ea7-4e6b-86b7-c1233548a4c1-plugins-config-volume" pod "babf1051-8ea7-4e6b-86b7-c1233548a4c1" (UID: "babf1051-8ea7-4e6b-86b7-c1233548a4c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:35.011127 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.011066    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babf1051-8ea7-4e6b-86b7-c1233548a4c1-kube-api-access-q6bb9" pod "babf1051-8ea7-4e6b-86b7-c1233548a4c1" (UID: "babf1051-8ea7-4e6b-86b7-c1233548a4c1"). InnerVolumeSpecName "kube-api-access-q6bb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:35.020646 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.020616    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"]
Oct 30 12:53:35.020873 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.020846    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy" containerID="cri-o://5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c" gracePeriod=30
Oct 30 12:53:35.105171 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.105139    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6bb9\" (UniqueName: \"kubernetes.io/projected/babf1051-8ea7-4e6b-86b7-c1233548a4c1-kube-api-access-q6bb9\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:35.105171 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.105168    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babf1051-8ea7-4e6b-86b7-c1233548a4c1-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:35.218547 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.218517    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"]
Oct 30 12:53:35.218766 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.218742    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2" podUID="415fb8df-7084-4bf4-9dd9-7920637accad" containerName="maas-api" containerID="cri-o://607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad" gracePeriod=30
Oct 30 12:53:35.366557 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.366512    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"]
Oct 30 12:53:35.367201 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.367173    4429 cpu_manager.go:414] "containerMap: removing container" podUID="babf1051-8ea7-4e6b-86b7-c1233548a4c1" containerName="payload-processing"
Oct 30 12:53:35.367201 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.367197    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babf1051-8ea7-4e6b-86b7-c1233548a4c1" containerName="payload-processing"
Oct 30 12:53:35.367398 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.367222    4429 cpu_manager.go:414] "containerMap: removing container" podUID="5275fd1a-2e74-4f2b-936c-5036f155b424" containerName="maas-api"
Oct 30 12:53:35.367398 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.367229    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5275fd1a-2e74-4f2b-936c-5036f155b424" containerName="maas-api"
Oct 30 12:53:35.367398 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.367354    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="5275fd1a-2e74-4f2b-936c-5036f155b424" containerName="maas-api"
Oct 30 12:53:35.367398 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.367375    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="babf1051-8ea7-4e6b-86b7-c1233548a4c1" containerName="payload-processing"
Oct 30 12:53:35.376312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.373624    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.390222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.390201    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-551dd1-openshift-default-dockercfg-mnncw\""
Oct 30 12:53:35.399074 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.399046    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"]
Oct 30 12:53:35.487135 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.487110    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:35.504647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.504594    4429 generic.go:356] "Generic (PLEG): container finished" podID="415fb8df-7084-4bf4-9dd9-7920637accad" containerID="607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad" exitCode=0
Oct 30 12:53:35.504858 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.504698    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2" event={"ID":"415fb8df-7084-4bf4-9dd9-7920637accad","Type":"ContainerDied","Data":"607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad"}
Oct 30 12:53:35.504858 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.504747    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2" event={"ID":"415fb8df-7084-4bf4-9dd9-7920637accad","Type":"ContainerDied","Data":"6cfcaec778fd5e4a1da6df6e15e64fbbee9375850be07dcafe37d8ebad68803c"}
Oct 30 12:53:35.504858 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.504781    4429 scope.go:122] "RemoveContainer" containerID="607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad"
Oct 30 12:53:35.505071 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.504922    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"
Oct 30 12:53:35.507790 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.507721    4429 generic.go:356] "Generic (PLEG): container finished" podID="babf1051-8ea7-4e6b-86b7-c1233548a4c1" containerID="83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91" exitCode=0
Oct 30 12:53:35.508222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.507940    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s" event={"ID":"babf1051-8ea7-4e6b-86b7-c1233548a4c1","Type":"ContainerDied","Data":"83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91"}
Oct 30 12:53:35.512258 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.512186    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s" event={"ID":"babf1051-8ea7-4e6b-86b7-c1233548a4c1","Type":"ContainerDied","Data":"07edfbba9b85ee933e12ebf4501fafb069463989a0798d314b1e1e73dc4cafb7"}
Oct 30 12:53:35.512510 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.508033    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"
Oct 30 12:53:35.513027 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.513005    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.513199 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.513181    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4fd\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-kube-api-access-kt4fd\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.513509 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.513487    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.513701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.513663    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.521535 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.521509    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.521934 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.521912    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.522105 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.522090    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.522259 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.522245    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.522531 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.522514    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.523396 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.523362    4429 scope.go:122] "RemoveContainer" containerID="607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad"
Oct 30 12:53:35.524080 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:35.524058    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad\": container with ID starting with 607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad not found: ID does not exist" containerID="607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad"
Oct 30 12:53:35.524255 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.524230    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad"} err="failed to get container status \"607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad\": rpc error: code = NotFound desc = could not find container \"607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad\": container with ID starting with 607a0a26c3f0b08f7109787417c99f938fd7c6a59a993e7cb14574095b4d3dad not found: ID does not exist"
Oct 30 12:53:35.524444 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.524432    4429 scope.go:122] "RemoveContainer" containerID="83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91"
Oct 30 12:53:35.540916 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.540890    4429 scope.go:122] "RemoveContainer" containerID="83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91"
Oct 30 12:53:35.541346 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:35.541323    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91\": container with ID starting with 83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91 not found: ID does not exist" containerID="83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91"
Oct 30 12:53:35.541493 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.541354    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91"} err="failed to get container status \"83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91\": rpc error: code = NotFound desc = could not find container \"83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91\": container with ID starting with 83cb6120f5bf40629762709b9afe465ea4dbed44e86645660fc19aba7fff1e91 not found: ID does not exist"
Oct 30 12:53:35.585168 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.585142    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"]
Oct 30 12:53:35.592511 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.592454    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fb8b5ba5-5fb4f495b6-4947s"]
Oct 30 12:53:35.624026 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624002    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415fb8df-7084-4bf4-9dd9-7920637accad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fb8df-7084-4bf4-9dd9-7920637accad-openshift-service-ca\") pod \"415fb8df-7084-4bf4-9dd9-7920637accad\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") "
Oct 30 12:53:35.624157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624066    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-metrics-certs\") pod \"415fb8df-7084-4bf4-9dd9-7920637accad\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") "
Oct 30 12:53:35.624157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624110    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415fb8df-7084-4bf4-9dd9-7920637accad-kube-api-access-77jpn\" (UniqueName: \"kubernetes.io/projected/415fb8df-7084-4bf4-9dd9-7920637accad-kube-api-access-77jpn\") pod \"415fb8df-7084-4bf4-9dd9-7920637accad\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") "
Oct 30 12:53:35.624270 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624232    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-maas-api-tls\") pod \"415fb8df-7084-4bf4-9dd9-7920637accad\" (UID: \"415fb8df-7084-4bf4-9dd9-7920637accad\") "
Oct 30 12:53:35.624490 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624466    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624490 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624480    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415fb8df-7084-4bf4-9dd9-7920637accad-openshift-service-ca" pod "415fb8df-7084-4bf4-9dd9-7920637accad" (UID: "415fb8df-7084-4bf4-9dd9-7920637accad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:35.624606 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624512    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624659 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624608    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624659 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624643    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624769 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624681    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624769 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624720    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624958 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624908    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624958 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624946    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624958 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624962    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4fd\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-kube-api-access-kt4fd\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.624958 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.624978    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.625328 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.625031    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.625328 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.625059    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.625328 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.625108    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fb8df-7084-4bf4-9dd9-7920637accad-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:35.625777 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.625753    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.625927 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.625821    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.627150 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.627126    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.627412 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.627384    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-metrics-certs" pod "415fb8df-7084-4bf4-9dd9-7920637accad" (UID: "415fb8df-7084-4bf4-9dd9-7920637accad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:35.627697 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.627677    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415fb8df-7084-4bf4-9dd9-7920637accad-kube-api-access-77jpn" pod "415fb8df-7084-4bf4-9dd9-7920637accad" (UID: "415fb8df-7084-4bf4-9dd9-7920637accad"). InnerVolumeSpecName "kube-api-access-77jpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:35.627785 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.627768    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-maas-api-tls" pod "415fb8df-7084-4bf4-9dd9-7920637accad" (UID: "415fb8df-7084-4bf4-9dd9-7920637accad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:35.628169 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.628150    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.650722 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.650678    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4fd\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-kube-api-access-kt4fd\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.651248 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.651213    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.700442 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.700418    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:35.726274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.726253    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:35.726447 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.726306    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jpn\" (UniqueName: \"kubernetes.io/projected/415fb8df-7084-4bf4-9dd9-7920637accad-kube-api-access-77jpn\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:35.726447 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.726324    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415fb8df-7084-4bf4-9dd9-7920637accad-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:35.860959 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.860827    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"]
Oct 30 12:53:35.867411 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:35.867306    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-873e4f54-59dc7894db-rhnw2"]
Oct 30 12:53:36.027217 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.027163    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"]
Oct 30 12:53:36.030347 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:36.030318    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcd3ba2_d0df_4a3d_ab08_18e952fe07b9.slice/crio-af9dc23b5f2f6b271d1a5d6aedafcbd057691ea89c1880df73f59f79585c9b30 WatchSource:0}: Error finding container af9dc23b5f2f6b271d1a5d6aedafcbd057691ea89c1880df73f59f79585c9b30: Status 404 returned error can't find the container with id af9dc23b5f2f6b271d1a5d6aedafcbd057691ea89c1880df73f59f79585c9b30
Oct 30 12:53:36.032857 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.032822    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:36.032974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.032901    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:36.032974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.032948    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:36.074157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.074132    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415fb8df-7084-4bf4-9dd9-7920637accad" path="/var/lib/kubelet/pods/415fb8df-7084-4bf4-9dd9-7920637accad/volumes"
Oct 30 12:53:36.074752 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.074713    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babf1051-8ea7-4e6b-86b7-c1233548a4c1" path="/var/lib/kubelet/pods/babf1051-8ea7-4e6b-86b7-c1233548a4c1/volumes"
Oct 30 12:53:36.516709 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.516675    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" event={"ID":"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9","Type":"ContainerStarted","Data":"d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460"}
Oct 30 12:53:36.517313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.517247    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" event={"ID":"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9","Type":"ContainerStarted","Data":"af9dc23b5f2f6b271d1a5d6aedafcbd057691ea89c1880df73f59f79585c9b30"}
Oct 30 12:53:36.567834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.567784    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" podStartSLOduration=1.567769192 podStartE2EDuration="1.567769192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:36.565003106 +0000 UTC m=+1274.620496941" watchObservedRunningTime="2026-10-30 12:53:36.567769192 +0000 UTC m=+1274.623263015"
Oct 30 12:53:36.613575 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.613545    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"]
Oct 30 12:53:36.614349 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.614093    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy" containerID="cri-o://d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b" gracePeriod=30
Oct 30 12:53:36.701397 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:36.701368    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:37.706661 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:37.706631    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:38.530219 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:38.530182    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:38.531207 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:38.531181    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:38.830395 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:38.830315    4429 patch_prober.go:29] interesting pod/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:53:38.830395 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:38.830370    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:53:40.284080 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.284030    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:53:40.375118 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375092    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istiod-ca-cert\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.375270 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375130    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-token\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-token\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.375270 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375254    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-credential-socket\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.375431 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375410    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-certs\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.375492 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375456    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-kube-api-access-nkrff\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-kube-api-access-nkrff\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.375545 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375480    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istiod-ca-cert" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:40.375600 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375572    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-credential-socket" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:40.375600 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375490    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-socket\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.375701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375670    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-socket" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:40.375754 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375674    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-podinfo\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.375754 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375740    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-data\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.375754 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375742    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-certs" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:40.375909 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375791    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-envoy\") pod \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\" (UID: \"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68\") "
Oct 30 12:53:40.376005 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.375986    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-data" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:40.376527 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.376179    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.376527 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.376251    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.376527 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.376266    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.376527 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.376306    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.376527 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.376328    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.378599 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.378571    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-kube-api-access-nkrff" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "kube-api-access-nkrff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:40.378969 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.378945    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-token" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:40.379153 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.379131    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-podinfo" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:53:40.379153 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.379140    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-envoy" pod "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" (UID: "9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:40.477208 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.477185    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkrff\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-kube-api-access-nkrff\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.477208 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.477208    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.477401 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.477216    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.477401 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.477225    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:40.540573 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.540509    4429 generic.go:356] "Generic (PLEG): container finished" podID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerID="5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c" exitCode=0
Oct 30 12:53:40.540692 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.540577    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" event={"ID":"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68","Type":"ContainerDied","Data":"5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c"}
Oct 30 12:53:40.540692 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.540604    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"
Oct 30 12:53:40.540692 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.540637    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj" event={"ID":"9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68","Type":"ContainerDied","Data":"7a59991e5846f18ab305ccbe3382fb08cca46ecd0208841cc69b26bf13a91f29"}
Oct 30 12:53:40.540692 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.540668    4429 scope.go:122] "RemoveContainer" containerID="5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c"
Oct 30 12:53:40.549831 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.549811    4429 scope.go:122] "RemoveContainer" containerID="0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d"
Oct 30 12:53:40.557952 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.557938    4429 scope.go:122] "RemoveContainer" containerID="5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c"
Oct 30 12:53:40.558206 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:40.558189    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c\": container with ID starting with 5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c not found: ID does not exist" containerID="5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c"
Oct 30 12:53:40.558252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.558216    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c"} err="failed to get container status \"5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c\": rpc error: code = NotFound desc = could not find container \"5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c\": container with ID starting with 5f479b2af7748e3fc3e5bd6be84c529c861c41670a2fc8c7201895efaf5d101c not found: ID does not exist"
Oct 30 12:53:40.558252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.558233    4429 scope.go:122] "RemoveContainer" containerID="0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d"
Oct 30 12:53:40.558554 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:40.558537    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d\": container with ID starting with 0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d not found: ID does not exist" containerID="0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d"
Oct 30 12:53:40.558595 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.558563    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d"} err="failed to get container status \"0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d\": rpc error: code = NotFound desc = could not find container \"0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d\": container with ID starting with 0c21f4fad5084b02d89e7e79b3da50065d3a53f37f17fe699a0cead6c760e36d not found: ID does not exist"
Oct 30 12:53:40.605571 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.605547    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"]
Oct 30 12:53:40.618342 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:40.618317    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f1c4045f-openshift-default-85f8967bbc-sgvkj"]
Oct 30 12:53:41.305952 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.305925    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"]
Oct 30 12:53:41.306347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306331    4429 cpu_manager.go:414] "containerMap: removing container" podUID="415fb8df-7084-4bf4-9dd9-7920637accad" containerName="maas-api"
Oct 30 12:53:41.306347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306343    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415fb8df-7084-4bf4-9dd9-7920637accad" containerName="maas-api"
Oct 30 12:53:41.306426 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306376    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy"
Oct 30 12:53:41.306426 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306381    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy"
Oct 30 12:53:41.306426 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306393    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy"
Oct 30 12:53:41.306426 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306398    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy"
Oct 30 12:53:41.306534 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306461    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy"
Oct 30 12:53:41.306534 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306469    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" containerName="istio-proxy"
Oct 30 12:53:41.306534 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.306474    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="415fb8df-7084-4bf4-9dd9-7920637accad" containerName="maas-api"
Oct 30 12:53:41.310925 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.310908    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:41.330048 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.330028    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-551dd1\""
Oct 30 12:53:41.330186 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.330035    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-551dd1-dockercfg-6rsbl\""
Oct 30 12:53:41.332669 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.332645    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"]
Oct 30 12:53:41.396959 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.396920    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3a2953-1200-438e-a224-6259a412c53f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2\" (UID: \"bf3a2953-1200-438e-a224-6259a412c53f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:41.397185 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.397023    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frf8j\" (UniqueName: \"kubernetes.io/projected/bf3a2953-1200-438e-a224-6259a412c53f-kube-api-access-frf8j\") pod \"payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2\" (UID: \"bf3a2953-1200-438e-a224-6259a412c53f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:41.497708 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.497646    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3a2953-1200-438e-a224-6259a412c53f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2\" (UID: \"bf3a2953-1200-438e-a224-6259a412c53f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:41.497907 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.497735    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frf8j\" (UniqueName: \"kubernetes.io/projected/bf3a2953-1200-438e-a224-6259a412c53f-kube-api-access-frf8j\") pod \"payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2\" (UID: \"bf3a2953-1200-438e-a224-6259a412c53f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:41.498402 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.498385    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3a2953-1200-438e-a224-6259a412c53f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2\" (UID: \"bf3a2953-1200-438e-a224-6259a412c53f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:41.560800 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.560737    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frf8j\" (UniqueName: \"kubernetes.io/projected/bf3a2953-1200-438e-a224-6259a412c53f-kube-api-access-frf8j\") pod \"payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2\" (UID: \"bf3a2953-1200-438e-a224-6259a412c53f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:41.623999 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.623966    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:41.876612 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.876587    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:41.922627 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:41.922593    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3a2953_1200_438e_a224_6259a412c53f.slice/crio-f5061124e200e0708d342e83c2fac37fe9f01c7261b6f7b8b0dd0bd1de608c59 WatchSource:0}: Error finding container f5061124e200e0708d342e83c2fac37fe9f01c7261b6f7b8b0dd0bd1de608c59: Status 404 returned error can't find the container with id f5061124e200e0708d342e83c2fac37fe9f01c7261b6f7b8b0dd0bd1de608c59
Oct 30 12:53:41.922740 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:41.922677    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"]
Oct 30 12:53:42.002227 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002204    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-data\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002330 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002234    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edc6de86-242b-4566-8d86-3e190fa55b61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc6de86-242b-4566-8d86-3e190fa55b61-istio-podinfo\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002330 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002319    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-istio-token\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-istio-token\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002406 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002376    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-certs\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002471 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002445    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-kube-api-access-k47d2\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-kube-api-access-k47d2\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002531 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002497    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-socket\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002531 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002498    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-data" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:42.002636 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002532    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc6de86-242b-4566-8d86-3e190fa55b61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc6de86-242b-4566-8d86-3e190fa55b61-istiod-ca-cert\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002636 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002584    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-envoy\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002736 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002649    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-certs" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:42.002786 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002732    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-credential-socket\") pod \"edc6de86-242b-4566-8d86-3e190fa55b61\" (UID: \"edc6de86-242b-4566-8d86-3e190fa55b61\") "
Oct 30 12:53:42.002786 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002773    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-socket" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:42.002920 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002891    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc6de86-242b-4566-8d86-3e190fa55b61-istiod-ca-cert" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:42.003025 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.002984    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-credential-socket" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:42.003523 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.003319    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.003523 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.003380    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.003523 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.003475    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.003523 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.003488    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc6de86-242b-4566-8d86-3e190fa55b61-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.003523 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.003501    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.004806 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.004782    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-envoy" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:42.004902 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.004805    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-kube-api-access-k47d2" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "kube-api-access-k47d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:42.004902 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.004860    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-istio-token" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:42.004902 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.004891    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edc6de86-242b-4566-8d86-3e190fa55b61-istio-podinfo" pod "edc6de86-242b-4566-8d86-3e190fa55b61" (UID: "edc6de86-242b-4566-8d86-3e190fa55b61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:53:42.073830 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.073798    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68" path="/var/lib/kubelet/pods/9a4fe4bf-7fa0-4c45-8472-4d1ca74b9e68/volumes"
Oct 30 12:53:42.104206 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.104144    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47d2\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-kube-api-access-k47d2\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.104206 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.104169    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc6de86-242b-4566-8d86-3e190fa55b61-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.104206 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.104182    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc6de86-242b-4566-8d86-3e190fa55b61-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.104206 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.104193    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc6de86-242b-4566-8d86-3e190fa55b61-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:42.371707 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:42.371630    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:53:42.552691 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.552636    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2" event={"ID":"bf3a2953-1200-438e-a224-6259a412c53f","Type":"ContainerStarted","Data":"7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35"}
Oct 30 12:53:42.552852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.552741    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2" event={"ID":"bf3a2953-1200-438e-a224-6259a412c53f","Type":"ContainerStarted","Data":"f5061124e200e0708d342e83c2fac37fe9f01c7261b6f7b8b0dd0bd1de608c59"}
Oct 30 12:53:42.552852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.552776    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:42.554515 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.554492    4429 generic.go:356] "Generic (PLEG): container finished" podID="edc6de86-242b-4566-8d86-3e190fa55b61" containerID="d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b" exitCode=0
Oct 30 12:53:42.554628 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.554534    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" event={"ID":"edc6de86-242b-4566-8d86-3e190fa55b61","Type":"ContainerDied","Data":"d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b"}
Oct 30 12:53:42.554628 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.554578    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf" event={"ID":"edc6de86-242b-4566-8d86-3e190fa55b61","Type":"ContainerDied","Data":"9f0e6faffdaef1f762b3715d28c5667ce1635e9351d2da4d076f768ecc119d21"}
Oct 30 12:53:42.554628 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.554599    4429 scope.go:122] "RemoveContainer" containerID="d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b"
Oct 30 12:53:42.554628 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.554546    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"
Oct 30 12:53:42.564727 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.564703    4429 scope.go:122] "RemoveContainer" containerID="d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1"
Oct 30 12:53:42.579328 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.579241    4429 scope.go:122] "RemoveContainer" containerID="d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b"
Oct 30 12:53:42.581547 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:42.581258    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b\": container with ID starting with d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b not found: ID does not exist" containerID="d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b"
Oct 30 12:53:42.581547 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.581324    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b"} err="failed to get container status \"d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b\": rpc error: code = NotFound desc = could not find container \"d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b\": container with ID starting with d078f19ddc1f59a3cddb756cec9af273e4d97053bcc1b04a38bd1521290ac40b not found: ID does not exist"
Oct 30 12:53:42.581547 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.581347    4429 scope.go:122] "RemoveContainer" containerID="d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1"
Oct 30 12:53:42.581757 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:42.581734    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1\": container with ID starting with d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1 not found: ID does not exist" containerID="d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1"
Oct 30 12:53:42.581808 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.581759    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1"} err="failed to get container status \"d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1\": rpc error: code = NotFound desc = could not find container \"d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1\": container with ID starting with d813c1379a8c60dbebb7e3d63bc04d7babef84247e3cb3b37052aabdfbf203f1 not found: ID does not exist"
Oct 30 12:53:42.598027 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.597987    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2" podStartSLOduration=1.597977097 podStartE2EDuration="1.597977097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:42.597227067 +0000 UTC m=+1280.652720890" watchObservedRunningTime="2026-10-30 12:53:42.597977097 +0000 UTC m=+1280.653470920"
Oct 30 12:53:42.638518 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.638467    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"]
Oct 30 12:53:42.642210 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:42.642192    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-873e4f54-openshift-default-5bd9df9c57-rhhjf"]
Oct 30 12:53:43.267171 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.267145    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"]
Oct 30 12:53:43.267590 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.267577    4429 cpu_manager.go:414] "containerMap: removing container" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy"
Oct 30 12:53:43.267647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.267591    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy"
Oct 30 12:53:43.267690 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.267670    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy"
Oct 30 12:53:43.267690 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.267681    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy"
Oct 30 12:53:43.272191 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.272173    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:43.290016 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.289983    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"]
Oct 30 12:53:43.290791 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.290770    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e652ca-dockercfg-c5xjw\""
Oct 30 12:53:43.290791 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.290775    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e652ca\""
Oct 30 12:53:43.364911 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.364877    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"]
Oct 30 12:53:43.365329 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.365316    4429 cpu_manager.go:414] "containerMap: removing container" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy"
Oct 30 12:53:43.365373 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.365331    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" containerName="istio-proxy"
Oct 30 12:53:43.368568 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.368553    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:43.390872 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.390844    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"]
Oct 30 12:53:43.417959 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.417909    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sck2\" (UniqueName: \"kubernetes.io/projected/03680b2d-91b9-40e8-a3c5-5061188362b4-kube-api-access-7sck2\") pod \"payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m\" (UID: \"03680b2d-91b9-40e8-a3c5-5061188362b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:43.418163 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.417987    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03680b2d-91b9-40e8-a3c5-5061188362b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m\" (UID: \"03680b2d-91b9-40e8-a3c5-5061188362b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:43.425985 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.425954    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"]
Oct 30 12:53:43.426225 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.426203    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d" podUID="f7bff32f-7499-41f2-8348-66bcfa97f0dc" containerName="maas-api" containerID="cri-o://4ae12d4ede84f9c0c34aa07255d39e17a211376955b23d8230cc60963a3e4682" gracePeriod=30
Oct 30 12:53:43.518712 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.518629    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfa2574-66ba-4023-b91f-37ed39e299e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc\" (UID: \"9dfa2574-66ba-4023-b91f-37ed39e299e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:43.518865 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.518754    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dgj\" (UniqueName: \"kubernetes.io/projected/9dfa2574-66ba-4023-b91f-37ed39e299e1-kube-api-access-42dgj\") pod \"payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc\" (UID: \"9dfa2574-66ba-4023-b91f-37ed39e299e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:43.518865 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.518831    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sck2\" (UniqueName: \"kubernetes.io/projected/03680b2d-91b9-40e8-a3c5-5061188362b4-kube-api-access-7sck2\") pod \"payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m\" (UID: \"03680b2d-91b9-40e8-a3c5-5061188362b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:43.518978 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.518878    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03680b2d-91b9-40e8-a3c5-5061188362b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m\" (UID: \"03680b2d-91b9-40e8-a3c5-5061188362b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:43.519600 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.519580    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03680b2d-91b9-40e8-a3c5-5061188362b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m\" (UID: \"03680b2d-91b9-40e8-a3c5-5061188362b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:43.549407 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.549385    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sck2\" (UniqueName: \"kubernetes.io/projected/03680b2d-91b9-40e8-a3c5-5061188362b4-kube-api-access-7sck2\") pod \"payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m\" (UID: \"03680b2d-91b9-40e8-a3c5-5061188362b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:43.561776 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.561743    4429 generic.go:356] "Generic (PLEG): container finished" podID="f7bff32f-7499-41f2-8348-66bcfa97f0dc" containerID="4ae12d4ede84f9c0c34aa07255d39e17a211376955b23d8230cc60963a3e4682" exitCode=0
Oct 30 12:53:43.561909 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.561817    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d" event={"ID":"f7bff32f-7499-41f2-8348-66bcfa97f0dc","Type":"ContainerDied","Data":"4ae12d4ede84f9c0c34aa07255d39e17a211376955b23d8230cc60963a3e4682"}
Oct 30 12:53:43.585336 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.585308    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:43.628955 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.627440    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfa2574-66ba-4023-b91f-37ed39e299e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc\" (UID: \"9dfa2574-66ba-4023-b91f-37ed39e299e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:43.628955 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.627610    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dgj\" (UniqueName: \"kubernetes.io/projected/9dfa2574-66ba-4023-b91f-37ed39e299e1-kube-api-access-42dgj\") pod \"payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc\" (UID: \"9dfa2574-66ba-4023-b91f-37ed39e299e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:43.628955 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.628853    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfa2574-66ba-4023-b91f-37ed39e299e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc\" (UID: \"9dfa2574-66ba-4023-b91f-37ed39e299e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:43.677165 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.677143    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dgj\" (UniqueName: \"kubernetes.io/projected/9dfa2574-66ba-4023-b91f-37ed39e299e1-kube-api-access-42dgj\") pod \"payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc\" (UID: \"9dfa2574-66ba-4023-b91f-37ed39e299e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:43.680835 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.680815    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:43.738984 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.738693    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:43.796043 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.795956    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"]
Oct 30 12:53:43.829398 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.829358    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bff32f-7499-41f2-8348-66bcfa97f0dc-kube-api-access-lx9mr\" (UniqueName: \"kubernetes.io/projected/f7bff32f-7499-41f2-8348-66bcfa97f0dc-kube-api-access-lx9mr\") pod \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") "
Oct 30 12:53:43.829576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.829453    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bff32f-7499-41f2-8348-66bcfa97f0dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bff32f-7499-41f2-8348-66bcfa97f0dc-openshift-service-ca\") pod \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") "
Oct 30 12:53:43.829576 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.829507    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-metrics-certs\") pod \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") "
Oct 30 12:53:43.829711 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.829640    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-maas-api-tls\") pod \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\" (UID: \"f7bff32f-7499-41f2-8348-66bcfa97f0dc\") "
Oct 30 12:53:43.829870 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.829844    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bff32f-7499-41f2-8348-66bcfa97f0dc-openshift-service-ca" pod "f7bff32f-7499-41f2-8348-66bcfa97f0dc" (UID: "f7bff32f-7499-41f2-8348-66bcfa97f0dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:43.830094 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.830073    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bff32f-7499-41f2-8348-66bcfa97f0dc-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:43.831796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.831772    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bff32f-7499-41f2-8348-66bcfa97f0dc-kube-api-access-lx9mr" pod "f7bff32f-7499-41f2-8348-66bcfa97f0dc" (UID: "f7bff32f-7499-41f2-8348-66bcfa97f0dc"). InnerVolumeSpecName "kube-api-access-lx9mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:43.832020 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.832004    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-maas-api-tls" pod "f7bff32f-7499-41f2-8348-66bcfa97f0dc" (UID: "f7bff32f-7499-41f2-8348-66bcfa97f0dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:43.832195 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.832161    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-metrics-certs" pod "f7bff32f-7499-41f2-8348-66bcfa97f0dc" (UID: "f7bff32f-7499-41f2-8348-66bcfa97f0dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:53:43.930359 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:43.930325    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03680b2d_91b9_40e8_a3c5_5061188362b4.slice/crio-cb285cce221354575797a5351f8f99bc431ad708283c1f12d6392f9cc566a106 WatchSource:0}: Error finding container cb285cce221354575797a5351f8f99bc431ad708283c1f12d6392f9cc566a106: Status 404 returned error can't find the container with id cb285cce221354575797a5351f8f99bc431ad708283c1f12d6392f9cc566a106
Oct 30 12:53:43.930506 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.930488    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"]
Oct 30 12:53:43.930608 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.930589    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:43.930648 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.930610    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7bff32f-7499-41f2-8348-66bcfa97f0dc-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:43.930648 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:43.930619    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9mr\" (UniqueName: \"kubernetes.io/projected/f7bff32f-7499-41f2-8348-66bcfa97f0dc-kube-api-access-lx9mr\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:44.009125 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.009101    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"]
Oct 30 12:53:44.009585 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:44.009559    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfa2574_66ba_4023_b91f_37ed39e299e1.slice/crio-13f9175ecbba7484293cad7dd45a648101869255567f7842c3b4452a881f64bd WatchSource:0}: Error finding container 13f9175ecbba7484293cad7dd45a648101869255567f7842c3b4452a881f64bd: Status 404 returned error can't find the container with id 13f9175ecbba7484293cad7dd45a648101869255567f7842c3b4452a881f64bd
Oct 30 12:53:44.073810 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.073783    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc6de86-242b-4566-8d86-3e190fa55b61" path="/var/lib/kubelet/pods/edc6de86-242b-4566-8d86-3e190fa55b61/volumes"
Oct 30 12:53:44.270308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.270254    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"]
Oct 30 12:53:44.270697 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.270684    4429 cpu_manager.go:414] "containerMap: removing container" podUID="f7bff32f-7499-41f2-8348-66bcfa97f0dc" containerName="maas-api"
Oct 30 12:53:44.270744 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.270700    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bff32f-7499-41f2-8348-66bcfa97f0dc" containerName="maas-api"
Oct 30 12:53:44.270786 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.270777    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bff32f-7499-41f2-8348-66bcfa97f0dc" containerName="maas-api"
Oct 30 12:53:44.274085 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.274064    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.295066 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.294963    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-e652ca-serving-cert\""
Oct 30 12:53:44.297245 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.297198    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"]
Oct 30 12:53:44.435701 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.435669    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hsc\" (UniqueName: \"kubernetes.io/projected/bc289b98-9450-489e-975a-b2d6ba7f5b9f-kube-api-access-w6hsc\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.436075 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.435717    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.436075 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.435868    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-metrics-certs\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.436075 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.435901    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc289b98-9450-489e-975a-b2d6ba7f5b9f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.536812 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.536774    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-metrics-certs\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.536988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.536841    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc289b98-9450-489e-975a-b2d6ba7f5b9f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.536988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.536922    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hsc\" (UniqueName: \"kubernetes.io/projected/bc289b98-9450-489e-975a-b2d6ba7f5b9f-kube-api-access-w6hsc\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.536988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.536966    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.537482 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.537460    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc289b98-9450-489e-975a-b2d6ba7f5b9f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.539220 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.539199    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-metrics-certs\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.539390 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.539374    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.563186 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.563102    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hsc\" (UniqueName: \"kubernetes.io/projected/bc289b98-9450-489e-975a-b2d6ba7f5b9f-kube-api-access-w6hsc\") pod \"maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.572259 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.572201    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d" event={"ID":"f7bff32f-7499-41f2-8348-66bcfa97f0dc","Type":"ContainerDied","Data":"2d35e9accc13b68a5d184b5afd6f723da296a1eee6e13160cdb3465a67f1ffc6"}
Oct 30 12:53:44.572541 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.572443    4429 scope.go:122] "RemoveContainer" containerID="4ae12d4ede84f9c0c34aa07255d39e17a211376955b23d8230cc60963a3e4682"
Oct 30 12:53:44.572656 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.572566    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"
Oct 30 12:53:44.578033 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.577990    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m" event={"ID":"03680b2d-91b9-40e8-a3c5-5061188362b4","Type":"ContainerStarted","Data":"8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac"}
Oct 30 12:53:44.578246 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.578217    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m" event={"ID":"03680b2d-91b9-40e8-a3c5-5061188362b4","Type":"ContainerStarted","Data":"cb285cce221354575797a5351f8f99bc431ad708283c1f12d6392f9cc566a106"}
Oct 30 12:53:44.578799 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.578750    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:44.586500 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.586470    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc" event={"ID":"9dfa2574-66ba-4023-b91f-37ed39e299e1","Type":"ContainerStarted","Data":"4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33"}
Oct 30 12:53:44.586706 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.586676    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc" event={"ID":"9dfa2574-66ba-4023-b91f-37ed39e299e1","Type":"ContainerStarted","Data":"13f9175ecbba7484293cad7dd45a648101869255567f7842c3b4452a881f64bd"}
Oct 30 12:53:44.587529 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.587503    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:44.589535 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.589513    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:44.641300 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.641231    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m" podStartSLOduration=1.641215244 podStartE2EDuration="1.641215244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:44.639663915 +0000 UTC m=+1282.695157739" watchObservedRunningTime="2026-10-30 12:53:44.641215244 +0000 UTC m=+1282.696709068"
Oct 30 12:53:44.685170 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.685113    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc" podStartSLOduration=1.685097405 podStartE2EDuration="1.685097405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:44.68345137 +0000 UTC m=+1282.738945189" watchObservedRunningTime="2026-10-30 12:53:44.685097405 +0000 UTC m=+1282.740591229"
Oct 30 12:53:44.722239 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.722205    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"]
Oct 30 12:53:44.726477 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.726453    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-98f58556-5dc49bcd8c-v4q8d"]
Oct 30 12:53:44.893272 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:44.893238    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc289b98_9450_489e_975a_b2d6ba7f5b9f.slice/crio-303580fbb4e6f23449f27dc2145ddea74efa57ff2413499f479c43e2c75c5403 WatchSource:0}: Error finding container 303580fbb4e6f23449f27dc2145ddea74efa57ff2413499f479c43e2c75c5403: Status 404 returned error can't find the container with id 303580fbb4e6f23449f27dc2145ddea74efa57ff2413499f479c43e2c75c5403
Oct 30 12:53:44.893272 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:44.893261    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"]
Oct 30 12:53:45.595064 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:45.594957    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh" event={"ID":"bc289b98-9450-489e-975a-b2d6ba7f5b9f","Type":"ContainerStarted","Data":"f4170b06343a224657bb52f90d8ad8e3e172c68bb31bd9f2933d0a7acbd04c85"}
Oct 30 12:53:45.595064 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:45.595031    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh" event={"ID":"bc289b98-9450-489e-975a-b2d6ba7f5b9f","Type":"ContainerStarted","Data":"303580fbb4e6f23449f27dc2145ddea74efa57ff2413499f479c43e2c75c5403"}
Oct 30 12:53:45.642511 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:45.642460    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh" podStartSLOduration=1.3326827190000001 podStartE2EDuration="1.642444257s" totalImagesPullingTime="309.761538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:45.64075436 +0000 UTC m=+1283.696248185" watchObservedRunningTime="2026-10-30 12:53:45.642444257 +0000 UTC m=+1283.697938110"
Oct 30 12:53:46.077983 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.077948    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bff32f-7499-41f2-8348-66bcfa97f0dc" path="/var/lib/kubelet/pods/f7bff32f-7499-41f2-8348-66bcfa97f0dc/volumes"
Oct 30 12:53:46.599829 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.599796    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:46.621594 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.621563    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"]
Oct 30 12:53:46.625551 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.625529    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:46.644343 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.644322    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7a8b03\""
Oct 30 12:53:46.677560 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.677373    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"]
Oct 30 12:53:46.782513 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.782472    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5prs\" (UniqueName: \"kubernetes.io/projected/b1b1df1d-e2a2-4fe2-8598-49573af73613-kube-api-access-z5prs\") pod \"payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft\" (UID: \"b1b1df1d-e2a2-4fe2-8598-49573af73613\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:46.795376 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.782962    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b1df1d-e2a2-4fe2-8598-49573af73613-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft\" (UID: \"b1b1df1d-e2a2-4fe2-8598-49573af73613\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:46.897164 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.897086    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5prs\" (UniqueName: \"kubernetes.io/projected/b1b1df1d-e2a2-4fe2-8598-49573af73613-kube-api-access-z5prs\") pod \"payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft\" (UID: \"b1b1df1d-e2a2-4fe2-8598-49573af73613\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:46.897312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.897200    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b1df1d-e2a2-4fe2-8598-49573af73613-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft\" (UID: \"b1b1df1d-e2a2-4fe2-8598-49573af73613\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:46.897906 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.897888    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b1df1d-e2a2-4fe2-8598-49573af73613-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft\" (UID: \"b1b1df1d-e2a2-4fe2-8598-49573af73613\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:46.925390 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.925358    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5prs\" (UniqueName: \"kubernetes.io/projected/b1b1df1d-e2a2-4fe2-8598-49573af73613-kube-api-access-z5prs\") pod \"payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft\" (UID: \"b1b1df1d-e2a2-4fe2-8598-49573af73613\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:46.939039 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:46.939012    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:47.302913 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:47.302884    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b1df1d_e2a2_4fe2_8598_49573af73613.slice/crio-1812a155c0da2189468727751168b1efd719b3004ad2c419753efe2475e312a1 WatchSource:0}: Error finding container 1812a155c0da2189468727751168b1efd719b3004ad2c419753efe2475e312a1: Status 404 returned error can't find the container with id 1812a155c0da2189468727751168b1efd719b3004ad2c419753efe2475e312a1
Oct 30 12:53:47.304953 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.304922    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"]
Oct 30 12:53:47.607533 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.606559    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft" event={"ID":"b1b1df1d-e2a2-4fe2-8598-49573af73613","Type":"ContainerStarted","Data":"eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66"}
Oct 30 12:53:47.607533 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.606610    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft" event={"ID":"b1b1df1d-e2a2-4fe2-8598-49573af73613","Type":"ContainerStarted","Data":"1812a155c0da2189468727751168b1efd719b3004ad2c419753efe2475e312a1"}
Oct 30 12:53:47.607533 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.606660    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:47.638717 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.638685    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"]
Oct 30 12:53:47.642382 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.642366    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:47.680313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.661270    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-bb86fd\""
Oct 30 12:53:47.684329 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.681686    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-bb86fd-dockercfg-5jm9w\""
Oct 30 12:53:47.688380 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.688350    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"]
Oct 30 12:53:47.690113 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.690055    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft" podStartSLOduration=1.690036744 podStartE2EDuration="1.690036744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:47.660992982 +0000 UTC m=+1285.716486811" watchObservedRunningTime="2026-10-30 12:53:47.690036744 +0000 UTC m=+1285.745530572"
Oct 30 12:53:47.705059 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.704535    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e10d3e-d608-45bb-bfc5-f8905b852890-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq\" (UID: \"94e10d3e-d608-45bb-bfc5-f8905b852890\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:47.706410 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.705659    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl9dz\" (UniqueName: \"kubernetes.io/projected/94e10d3e-d608-45bb-bfc5-f8905b852890-kube-api-access-xl9dz\") pod \"payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq\" (UID: \"94e10d3e-d608-45bb-bfc5-f8905b852890\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:47.808247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.807736    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e10d3e-d608-45bb-bfc5-f8905b852890-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq\" (UID: \"94e10d3e-d608-45bb-bfc5-f8905b852890\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:47.808247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.807900    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl9dz\" (UniqueName: \"kubernetes.io/projected/94e10d3e-d608-45bb-bfc5-f8905b852890-kube-api-access-xl9dz\") pod \"payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq\" (UID: \"94e10d3e-d608-45bb-bfc5-f8905b852890\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:47.809660 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.809628    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e10d3e-d608-45bb-bfc5-f8905b852890-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq\" (UID: \"94e10d3e-d608-45bb-bfc5-f8905b852890\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:47.825562 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.825534    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"]
Oct 30 12:53:47.829181 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.829166    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:47.834920 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.834896    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl9dz\" (UniqueName: \"kubernetes.io/projected/94e10d3e-d608-45bb-bfc5-f8905b852890-kube-api-access-xl9dz\") pod \"payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq\" (UID: \"94e10d3e-d608-45bb-bfc5-f8905b852890\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:47.876388 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.864587    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7a8b03-serving-cert\""
Oct 30 12:53:47.877838 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.877806    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"]
Oct 30 12:53:47.908739 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.908705    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mcl\" (UniqueName: \"kubernetes.io/projected/22b81bf5-5279-44b4-a512-a1641b4f6a62-kube-api-access-l4mcl\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:47.908919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.908774    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-metrics-certs\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:47.908919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.908867    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b81bf5-5279-44b4-a512-a1641b4f6a62-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:47.908919 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.908914    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:47.995761 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:47.995545    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:48.010636 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.010216    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mcl\" (UniqueName: \"kubernetes.io/projected/22b81bf5-5279-44b4-a512-a1641b4f6a62-kube-api-access-l4mcl\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.010636 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.010256    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-metrics-certs\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.010636 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.010316    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b81bf5-5279-44b4-a512-a1641b4f6a62-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.010636 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.010349    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.011234 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.011209    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b81bf5-5279-44b4-a512-a1641b4f6a62-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.013182 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.013140    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.013633 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.013611    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-metrics-certs\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.035216 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.035188    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mcl\" (UniqueName: \"kubernetes.io/projected/22b81bf5-5279-44b4-a512-a1641b4f6a62-kube-api-access-l4mcl\") pod \"maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.142334 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.142230    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:48.334874 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.334844    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"]
Oct 30 12:53:48.335004 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:48.334980    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e10d3e_d608_45bb_bfc5_f8905b852890.slice/crio-83517657f9e84e8a06766713cbc5c1597262ae42a00f7fd33d2d259299e51e05 WatchSource:0}: Error finding container 83517657f9e84e8a06766713cbc5c1597262ae42a00f7fd33d2d259299e51e05: Status 404 returned error can't find the container with id 83517657f9e84e8a06766713cbc5c1597262ae42a00f7fd33d2d259299e51e05
Oct 30 12:53:48.497915 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.490613    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"]
Oct 30 12:53:48.498239 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:48.498205    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b81bf5_5279_44b4_a512_a1641b4f6a62.slice/crio-5cda3c055452c98012d5306398db467c67ba365b33a504b5dadcb52b4fe2be0a WatchSource:0}: Error finding container 5cda3c055452c98012d5306398db467c67ba365b33a504b5dadcb52b4fe2be0a: Status 404 returned error can't find the container with id 5cda3c055452c98012d5306398db467c67ba365b33a504b5dadcb52b4fe2be0a
Oct 30 12:53:48.612338 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.612297    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs" event={"ID":"22b81bf5-5279-44b4-a512-a1641b4f6a62","Type":"ContainerStarted","Data":"5cda3c055452c98012d5306398db467c67ba365b33a504b5dadcb52b4fe2be0a"}
Oct 30 12:53:48.614031 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.613995    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq" event={"ID":"94e10d3e-d608-45bb-bfc5-f8905b852890","Type":"ContainerStarted","Data":"3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b"}
Oct 30 12:53:48.614166 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.614046    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq" event={"ID":"94e10d3e-d608-45bb-bfc5-f8905b852890","Type":"ContainerStarted","Data":"83517657f9e84e8a06766713cbc5c1597262ae42a00f7fd33d2d259299e51e05"}
Oct 30 12:53:48.614166 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.614098    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:48.796395 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:48.796350    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm" podUID="c729fc62-18ca-4d3a-9887-b7a77537603d" containerName="payload-pre-processing" containerID="cri-o://e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb" gracePeriod=25
Oct 30 12:53:49.268863 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.268833    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:49.346151 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.346101    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq" podStartSLOduration=2.346085311 podStartE2EDuration="2.346085311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:48.669129295 +0000 UTC m=+1286.724623120" watchObservedRunningTime="2026-10-30 12:53:49.346085311 +0000 UTC m=+1287.401579135"
Oct 30 12:53:49.358326 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.347045    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c729fc62-18ca-4d3a-9887-b7a77537603d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c729fc62-18ca-4d3a-9887-b7a77537603d-plugins-config-volume\") pod \"c729fc62-18ca-4d3a-9887-b7a77537603d\" (UID: \"c729fc62-18ca-4d3a-9887-b7a77537603d\") "
Oct 30 12:53:49.358326 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.347085    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c729fc62-18ca-4d3a-9887-b7a77537603d-kube-api-access-2vgh7\" (UniqueName: \"kubernetes.io/projected/c729fc62-18ca-4d3a-9887-b7a77537603d-kube-api-access-2vgh7\") pod \"c729fc62-18ca-4d3a-9887-b7a77537603d\" (UID: \"c729fc62-18ca-4d3a-9887-b7a77537603d\") "
Oct 30 12:53:49.358326 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.354540    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c729fc62-18ca-4d3a-9887-b7a77537603d-plugins-config-volume" pod "c729fc62-18ca-4d3a-9887-b7a77537603d" (UID: "c729fc62-18ca-4d3a-9887-b7a77537603d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:53:49.365323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.358869    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c729fc62-18ca-4d3a-9887-b7a77537603d-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:49.379471 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.379430    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c729fc62-18ca-4d3a-9887-b7a77537603d-kube-api-access-2vgh7" pod "c729fc62-18ca-4d3a-9887-b7a77537603d" (UID: "c729fc62-18ca-4d3a-9887-b7a77537603d"). InnerVolumeSpecName "kube-api-access-2vgh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:49.462704 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.460007    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vgh7\" (UniqueName: \"kubernetes.io/projected/c729fc62-18ca-4d3a-9887-b7a77537603d-kube-api-access-2vgh7\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:53:49.620909 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.620878    4429 generic.go:356] "Generic (PLEG): container finished" podID="c729fc62-18ca-4d3a-9887-b7a77537603d" containerID="e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb" exitCode=0
Oct 30 12:53:49.621375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.620927    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"
Oct 30 12:53:49.621375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.620961    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm" event={"ID":"c729fc62-18ca-4d3a-9887-b7a77537603d","Type":"ContainerDied","Data":"e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb"}
Oct 30 12:53:49.621375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.621037    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm" event={"ID":"c729fc62-18ca-4d3a-9887-b7a77537603d","Type":"ContainerDied","Data":"37e8c545d30f1bab041ff8634ce33e2d9b65511db8250c20b72f50cceeacde3b"}
Oct 30 12:53:49.621375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.621060    4429 scope.go:122] "RemoveContainer" containerID="e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb"
Oct 30 12:53:49.624983 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.624963    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs" event={"ID":"22b81bf5-5279-44b4-a512-a1641b4f6a62","Type":"ContainerStarted","Data":"598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b"}
Oct 30 12:53:49.625126 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.625109    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:49.632680 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.632660    4429 scope.go:122] "RemoveContainer" containerID="e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb"
Oct 30 12:53:49.632947 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:53:49.632929    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb\": container with ID starting with e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb not found: ID does not exist" containerID="e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb"
Oct 30 12:53:49.633000 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.632956    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb"} err="failed to get container status \"e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb\": rpc error: code = NotFound desc = could not find container \"e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb\": container with ID starting with e2cb418a06352e5a697b3df9deab3bd5e06a5a213afbb198d6ac99cd003e39bb not found: ID does not exist"
Oct 30 12:53:49.693391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.693362    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"]
Oct 30 12:53:49.696398 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.696374    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-98f58556-75bd57c5c5-mnztm"]
Oct 30 12:53:49.775362 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:49.774457    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs" podStartSLOduration=2.079150258 podStartE2EDuration="2.774436696s" totalImagesPullingTime="695.286438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:49.772725892 +0000 UTC m=+1287.828219718" watchObservedRunningTime="2026-10-30 12:53:49.774436696 +0000 UTC m=+1287.829930523"
Oct 30 12:53:50.044046 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.043968    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"]
Oct 30 12:53:50.044569 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.044553    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c729fc62-18ca-4d3a-9887-b7a77537603d" containerName="payload-pre-processing"
Oct 30 12:53:50.044623 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.044571    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c729fc62-18ca-4d3a-9887-b7a77537603d" containerName="payload-pre-processing"
Oct 30 12:53:50.044684 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.044670    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c729fc62-18ca-4d3a-9887-b7a77537603d" containerName="payload-pre-processing"
Oct 30 12:53:50.054579 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.054548    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.080663 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.078437    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-bb86fd-serving-cert\""
Oct 30 12:53:50.084234 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.083774    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c729fc62-18ca-4d3a-9887-b7a77537603d" path="/var/lib/kubelet/pods/c729fc62-18ca-4d3a-9887-b7a77537603d/volumes"
Oct 30 12:53:50.084234 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.084163    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"]
Oct 30 12:53:50.175819 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.175785    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b7fde-8638-4360-b709-58c7f03ca5e2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.176005 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.175836    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9g7v\" (UniqueName: \"kubernetes.io/projected/3c5b7fde-8638-4360-b709-58c7f03ca5e2-kube-api-access-m9g7v\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.176005 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.175859    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.176146 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.175997    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-metrics-certs\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.278699 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.277934    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b7fde-8638-4360-b709-58c7f03ca5e2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.278699 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.278005    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9g7v\" (UniqueName: \"kubernetes.io/projected/3c5b7fde-8638-4360-b709-58c7f03ca5e2-kube-api-access-m9g7v\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.278699 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.278039    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.278699 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.278084    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-metrics-certs\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.280534 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.280499    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b7fde-8638-4360-b709-58c7f03ca5e2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.283439 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.283412    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.286520 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.286432    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-metrics-certs\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.308972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.308905    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9g7v\" (UniqueName: \"kubernetes.io/projected/3c5b7fde-8638-4360-b709-58c7f03ca5e2-kube-api-access-m9g7v\") pod \"maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.384927 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.384895    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:50.719706 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:50.719683    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"]
Oct 30 12:53:50.720446 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:53:50.720425    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5b7fde_8638_4360_b709_58c7f03ca5e2.slice/crio-22e82cc4dad37bbb5997ab6d6a346ab05f1b5faeffef9a0cc391ff15827362ec WatchSource:0}: Error finding container 22e82cc4dad37bbb5997ab6d6a346ab05f1b5faeffef9a0cc391ff15827362ec: Status 404 returned error can't find the container with id 22e82cc4dad37bbb5997ab6d6a346ab05f1b5faeffef9a0cc391ff15827362ec
Oct 30 12:53:51.639795 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:51.639755    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx" event={"ID":"3c5b7fde-8638-4360-b709-58c7f03ca5e2","Type":"ContainerStarted","Data":"b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6"}
Oct 30 12:53:51.639948 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:51.639819    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx" event={"ID":"3c5b7fde-8638-4360-b709-58c7f03ca5e2","Type":"ContainerStarted","Data":"22e82cc4dad37bbb5997ab6d6a346ab05f1b5faeffef9a0cc391ff15827362ec"}
Oct 30 12:53:51.640006 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:51.639987    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:51.693320 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:51.693238    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx" podStartSLOduration=1.408129333 podStartE2EDuration="1.693215554s" totalImagesPullingTime="285.086221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:51.69151607 +0000 UTC m=+1289.747009894" watchObservedRunningTime="2026-10-30 12:53:51.693215554 +0000 UTC m=+1289.748709379"
Oct 30 12:53:52.614145 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:52.614118    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 12:53:53.565801 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:53.565775    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 12:53:54.203081 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:54.202246    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:46414->10.133.0.41:15021: read: connection reset by peer" start-of-body=
Oct 30 12:53:54.203081 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:54.202347    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:46414->10.133.0.41:15021: read: connection reset by peer"
Oct 30 12:53:54.659113 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:54.659028    4429 generic.go:356] "Generic (PLEG): container finished" podID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerID="d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460" exitCode=0
Oct 30 12:53:54.659327 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:54.659128    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" event={"ID":"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9","Type":"ContainerDied","Data":"d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460"}
Oct 30 12:53:54.659667 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:54.659646    4429 scope.go:122] "RemoveContainer" containerID="d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460"
Oct 30 12:53:54.660986 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:54.660961    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:54.661044 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:54.661027    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:54.661080 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:54.661059    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:53:55.599069 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:55.599043    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 12:53:55.639482 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:55.639456    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 12:53:55.691592 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:55.691561    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" event={"ID":"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9","Type":"ContainerStarted","Data":"4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635"}
Oct 30 12:53:55.701465 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:55.701437    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:53:55.701651 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:55.701617    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:53:55.701734 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:55.701649    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:53:56.602714 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:56.602684    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 12:53:56.701643 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:56.701611    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:53:56.701801 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:56.701658    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:53:57.678008 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:57.677914    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 12:53:57.701413 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:57.701377    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:53:57.703001 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:57.701440    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:53:58.618128 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:58.618098    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 12:53:58.700878 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:58.700850    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:53:58.701242 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:58.700903    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:53:59.628925 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:59.628899    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 12:53:59.701345 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:59.701309    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:53:59.701799 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:53:59.701387    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 12:54:00.828913 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:00.828881    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:54:00.829401 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:00.829320    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:54:00.829979 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:00.829958    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 12:54:12.371879 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:54:12.371800    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:54:27.561627 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.561591    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"]
Oct 30 12:54:27.567974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.567952    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.584321 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.584237    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"]
Oct 30 12:54:27.595593 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.595568    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"default-dockercfg-xk2th\""
Oct 30 12:54:27.595709 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.595597    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"simulator-w1-6ff0e6-kserve-self-signed-certs\""
Oct 30 12:54:27.598253 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.598235    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"openshift-service-ca.crt\""
Oct 30 12:54:27.598540 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.598521    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"kube-root-ca.crt\""
Oct 30 12:54:27.662366 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.662341    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-tmp-dir\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.662502 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.662408    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-model-cache\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.662570 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.662530    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-dshm\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.662621 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.662563    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-home\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.662657 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.662633    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phlvn\" (UniqueName: \"kubernetes.io/projected/1494f395-8b0b-472c-bf57-645835078d44-kube-api-access-phlvn\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.662697 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.662690    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1494f395-8b0b-472c-bf57-645835078d44-tls-certs\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764002 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.763974    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-tmp-dir\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764116 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.764028    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-model-cache\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764116 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.764063    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-dshm\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764116 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.764079    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-home\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764116 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.764102    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phlvn\" (UniqueName: \"kubernetes.io/projected/1494f395-8b0b-472c-bf57-645835078d44-kube-api-access-phlvn\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764265 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.764140    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1494f395-8b0b-472c-bf57-645835078d44-tls-certs\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764431 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.764410    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-tmp-dir\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764522 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.764477    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-model-cache\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.764570 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.764552    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-home\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.766389 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.766368    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-dshm\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.766541 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.766525    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1494f395-8b0b-472c-bf57-645835078d44-tls-certs\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.802487 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.802403    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phlvn\" (UniqueName: \"kubernetes.io/projected/1494f395-8b0b-472c-bf57-645835078d44-kube-api-access-phlvn\") pod \"simulator-w1-6ff0e6-kserve-86878c6495-mklxs\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") " pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:27.880574 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:27.880500    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:28.176086 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:28.176058    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"]
Oct 30 12:54:28.176766 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:54:28.176743    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1494f395_8b0b_472c_bf57_645835078d44.slice/crio-8b9c6093f1cd0ccd82765ad1a714269d5b20c671b5b22f3716aebd91dac71a43 WatchSource:0}: Error finding container 8b9c6093f1cd0ccd82765ad1a714269d5b20c671b5b22f3716aebd91dac71a43: Status 404 returned error can't find the container with id 8b9c6093f1cd0ccd82765ad1a714269d5b20c671b5b22f3716aebd91dac71a43
Oct 30 12:54:28.879436 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:28.879399    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" event={"ID":"1494f395-8b0b-472c-bf57-645835078d44","Type":"ContainerStarted","Data":"faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790"}
Oct 30 12:54:28.879917 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:28.879445    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" event={"ID":"1494f395-8b0b-472c-bf57-645835078d44","Type":"ContainerStarted","Data":"8b9c6093f1cd0ccd82765ad1a714269d5b20c671b5b22f3716aebd91dac71a43"}
Oct 30 12:54:28.880697 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:28.880673    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:28.928071 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:28.928014    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podStartSLOduration=1.927999204 podStartE2EDuration="1.927999204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:28.926618789 +0000 UTC m=+1326.982112641" watchObservedRunningTime="2026-10-30 12:54:28.927999204 +0000 UTC m=+1326.983493059"
Oct 30 12:54:37.880694 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:37.880651    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:37.893396 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:37.893370    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:37.944619 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:37.944589    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:54:38.039059 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.039028    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"]
Oct 30 12:54:38.043237 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.043214    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.061441 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.061419    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"openshift-service-ca.crt\""
Oct 30 12:54:38.061634 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.061436    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"default-dockercfg-pdp25\""
Oct 30 12:54:38.061634 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.061521    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"premium-w4-e652ca-kserve-self-signed-certs\""
Oct 30 12:54:38.061634 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.061580    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"kube-root-ca.crt\""
Oct 30 12:54:38.064413 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.064393    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"]
Oct 30 12:54:38.172270 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.172202    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-dshm\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.172270 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.172248    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-home\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.172270 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.172269    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chntf\" (UniqueName: \"kubernetes.io/projected/f271de4a-9331-4b94-8d60-24a3eeb2df0c-kube-api-access-chntf\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.172613 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.172423    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tmp-dir\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.172613 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.172465    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tls-certs\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.172613 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.172486    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-model-cache\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.273746 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.273711    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tmp-dir\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.273944 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.273759    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tls-certs\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.273944 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.273794    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-model-cache\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.273944 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.273825    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-dshm\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.274123 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.273967    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-home\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.274123 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.274016    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chntf\" (UniqueName: \"kubernetes.io/projected/f271de4a-9331-4b94-8d60-24a3eeb2df0c-kube-api-access-chntf\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.274229 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.274205    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-model-cache\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.274310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.274266    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tmp-dir\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.274386 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.274369    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-home\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.276198 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.276177    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-dshm\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.276493 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.276477    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tls-certs\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.299005 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.298978    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chntf\" (UniqueName: \"kubernetes.io/projected/f271de4a-9331-4b94-8d60-24a3eeb2df0c-kube-api-access-chntf\") pod \"premium-w4-e652ca-kserve-79d85d4446-v547x\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") " pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.358604 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.358580    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:38.657520 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:54:38.657482    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf271de4a_9331_4b94_8d60_24a3eeb2df0c.slice/crio-fe6260bce3f259ffa8e5ceb374139465403a0fc3c237181cda234c225b6b42a6 WatchSource:0}: Error finding container fe6260bce3f259ffa8e5ceb374139465403a0fc3c237181cda234c225b6b42a6: Status 404 returned error can't find the container with id fe6260bce3f259ffa8e5ceb374139465403a0fc3c237181cda234c225b6b42a6
Oct 30 12:54:38.657936 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.657915    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"]
Oct 30 12:54:38.940165 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.940129    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" event={"ID":"f271de4a-9331-4b94-8d60-24a3eeb2df0c","Type":"ContainerStarted","Data":"c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b"}
Oct 30 12:54:38.940165 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.940168    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" event={"ID":"f271de4a-9331-4b94-8d60-24a3eeb2df0c","Type":"ContainerStarted","Data":"fe6260bce3f259ffa8e5ceb374139465403a0fc3c237181cda234c225b6b42a6"}
Oct 30 12:54:38.992868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:38.992818    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podStartSLOduration=0.992802747 podStartE2EDuration="992.802747ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:38.991117404 +0000 UTC m=+1337.046611231" watchObservedRunningTime="2026-10-30 12:54:38.992802747 +0000 UTC m=+1337.048296608"
Oct 30 12:54:39.358801 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:39.358707    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:42.371597 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:54:42.371545    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:54:48.359021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:48.358988    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:48.371404 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:48.371385    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:49.002688 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:49.002664    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:54:59.475597 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.475561    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"]
Oct 30 12:54:59.481216 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.481191    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.505597 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.505570    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"]
Oct 30 12:54:59.522745 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.522724    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"premium-w1-6ff0e6-kserve-self-signed-certs\""
Oct 30 12:54:59.574801 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.574776    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-model-cache\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.574916 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.574820    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tmp-dir\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.574965 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.574923    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-dshm\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.575015 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.574982    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2knk\" (UniqueName: \"kubernetes.io/projected/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-kube-api-access-s2knk\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.575089 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.575071    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-home\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.575153 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.575141    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tls-certs\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676139 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676113    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-model-cache\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676166    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tmp-dir\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676252 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676225    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-dshm\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676400 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676276    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knk\" (UniqueName: \"kubernetes.io/projected/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-kube-api-access-s2knk\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676400 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676323    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-home\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676400 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676381    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tls-certs\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676563 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676507    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-model-cache\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676625 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676598    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tmp-dir\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.676763 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.676735    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-home\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.678687 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.678661    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-dshm\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.678912 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.678895    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tls-certs\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.706294 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.706230    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2knk\" (UniqueName: \"kubernetes.io/projected/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-kube-api-access-s2knk\") pod \"premium-w1-6ff0e6-kserve-646d49b58-ncntd\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") " pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:54:59.796276 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:54:59.796198    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:55:00.094666 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:00.094637    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"]
Oct 30 12:55:00.095065 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:55:00.095037    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fd7c4dc_0253_42f2_a62d_3af6b6de9cf1.slice/crio-244a9fc334a00fb7f71d4081953d4e83132b2fc3582ac2a32f9d8c45e2ca99d7 WatchSource:0}: Error finding container 244a9fc334a00fb7f71d4081953d4e83132b2fc3582ac2a32f9d8c45e2ca99d7: Status 404 returned error can't find the container with id 244a9fc334a00fb7f71d4081953d4e83132b2fc3582ac2a32f9d8c45e2ca99d7
Oct 30 12:55:01.053059 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:01.053027    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" event={"ID":"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1","Type":"ContainerStarted","Data":"f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a"}
Oct 30 12:55:01.053492 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:01.053069    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" event={"ID":"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1","Type":"ContainerStarted","Data":"244a9fc334a00fb7f71d4081953d4e83132b2fc3582ac2a32f9d8c45e2ca99d7"}
Oct 30 12:55:01.099361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:01.099314    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podStartSLOduration=2.099298904 podStartE2EDuration="2.099298904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:01.097871282 +0000 UTC m=+1359.153365107" watchObservedRunningTime="2026-10-30 12:55:01.099298904 +0000 UTC m=+1359.154792729"
Oct 30 12:55:01.797110 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:01.797068    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:55:06.807430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.807399    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"]
Oct 30 12:55:06.811120 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.811098    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.832398 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.832372    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e652ca-openshift-default-dockercfg-rr2pm\""
Oct 30 12:55:06.847147 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.847120    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"]
Oct 30 12:55:06.946328 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946272    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wx5\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-kube-api-access-27wx5\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.946488 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946386    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.946488 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946456    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.946583 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946497    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.946637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946586    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed00f65-2134-46d3-9505-c12c7171eb33-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.946705 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946684    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed00f65-2134-46d3-9505-c12c7171eb33-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.946762 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946742    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.946814 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946802    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:06.946852 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:06.946830    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.047731 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.047703    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wx5\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-kube-api-access-27wx5\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.047731 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.047731    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.047900 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.047753    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.047900 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.047790    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.047900 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.047827    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed00f65-2134-46d3-9505-c12c7171eb33-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.047900 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.047882    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed00f65-2134-46d3-9505-c12c7171eb33-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.048095 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.047925    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.048095 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.047985    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.048095 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.048023    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.048262 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.048235    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.048387 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.048365    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.048439 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.048400    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.048480 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.048443    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.048596 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.048579    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed00f65-2134-46d3-9505-c12c7171eb33-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.050451 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.050429    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.050599 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.050582    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed00f65-2134-46d3-9505-c12c7171eb33-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.075746 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.075699    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.075816 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.075757    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wx5\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-kube-api-access-27wx5\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.124203 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.124183    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:07.444365 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.444338    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"]
Oct 30 12:55:07.444569 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:55:07.444544    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed00f65_2134_46d3_9505_c12c7171eb33.slice/crio-e6eb0d56469ce93f17087fcd242c061968f56048e5af266a8b353b9a704c2f51 WatchSource:0}: Error finding container e6eb0d56469ce93f17087fcd242c061968f56048e5af266a8b353b9a704c2f51: Status 404 returned error can't find the container with id e6eb0d56469ce93f17087fcd242c061968f56048e5af266a8b353b9a704c2f51
Oct 30 12:55:07.446743 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.446715    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:55:07.446841 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.446772    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:55:07.446841 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:07.446798    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:55:08.090777 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:08.090746    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" event={"ID":"aed00f65-2134-46d3-9505-c12c7171eb33","Type":"ContainerStarted","Data":"0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea"}
Oct 30 12:55:08.091160 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:08.090802    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" event={"ID":"aed00f65-2134-46d3-9505-c12c7171eb33","Type":"ContainerStarted","Data":"e6eb0d56469ce93f17087fcd242c061968f56048e5af266a8b353b9a704c2f51"}
Oct 30 12:55:08.124973 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:08.124947    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:08.126535 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:08.126517    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:55:08.126593 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:08.126558    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:55:09.124853 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:09.124823    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:55:09.125337 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:09.124874    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:55:09.796732 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:09.796701    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:55:09.809868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:09.809839    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:55:09.879057 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:09.879006    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podStartSLOduration=3.878990082 podStartE2EDuration="3.878990082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:08.1734115 +0000 UTC m=+1366.228905349" watchObservedRunningTime="2026-10-30 12:55:09.878990082 +0000 UTC m=+1367.934483906"
Oct 30 12:55:10.111529 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:10.111447    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:55:10.125049 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:10.125018    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:55:10.125435 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:10.125067    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:55:11.124663 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:11.124637    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:55:11.124839 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:11.124683    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:55:12.125419 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:12.125391    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:55:12.125801 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:12.125462    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:55:12.395217 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:55:12.395139    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:55:13.124687 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:13.124656    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 12:55:13.124872 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:13.124720    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 12:55:14.129536 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:14.129511    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:14.130014 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:14.129954    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:14.130717 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:55:14.130690    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 12:55:42.372907 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:55:42.372820    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:56:08.449781 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.449747    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"]
Oct 30 12:56:08.495775 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.495742    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"]
Oct 30 12:56:08.496157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.496056    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.537307 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.515592    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7a8b03-openshift-default-dockercfg-mbrk9\""
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.713549    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.713619    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.713697    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698f0851-42a2-4cf5-abfd-676cefceeef1-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.713752    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.713813    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.713846    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rckm5\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-kube-api-access-rckm5\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.713994    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.714035    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.721796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.714088    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.821311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.816882    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.830408 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.830380    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.834307 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.830606    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.834534 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.834494    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.835329 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.834658    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.836410 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.836384    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698f0851-42a2-4cf5-abfd-676cefceeef1-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.836578 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.836558    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.836727 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.836711    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.836846 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.836831    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckm5\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-kube-api-access-rckm5\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.837325 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.834906    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.837757 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.837736    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.837885 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.835332    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.837988 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.835603    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.838084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.838037    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698f0851-42a2-4cf5-abfd-676cefceeef1-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.839803 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.839760    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.841604 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.841552    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.885124 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.885033    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckm5\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-kube-api-access-rckm5\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:08.885831 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:08.885808    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:09.036554 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:09.036472    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:09.388041 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:09.388009    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"]
Oct 30 12:56:09.389438 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:56:09.389406    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698f0851_42a2_4cf5_abfd_676cefceeef1.slice/crio-d5ffac9b5c4dbc4aa026d201c6eeaf15f7e70a25276a8a772736405318ede0c2 WatchSource:0}: Error finding container d5ffac9b5c4dbc4aa026d201c6eeaf15f7e70a25276a8a772736405318ede0c2: Status 404 returned error can't find the container with id d5ffac9b5c4dbc4aa026d201c6eeaf15f7e70a25276a8a772736405318ede0c2
Oct 30 12:56:09.392490 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:09.392451    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:09.392584 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:09.392537    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:09.392584 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:09.392579    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:09.478713 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:09.478550    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" event={"ID":"698f0851-42a2-4cf5-abfd-676cefceeef1","Type":"ContainerStarted","Data":"15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb"}
Oct 30 12:56:09.478713 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:09.478608    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" event={"ID":"698f0851-42a2-4cf5-abfd-676cefceeef1","Type":"ContainerStarted","Data":"d5ffac9b5c4dbc4aa026d201c6eeaf15f7e70a25276a8a772736405318ede0c2"}
Oct 30 12:56:09.540167 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:09.540117    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" podStartSLOduration=1.540099213 podStartE2EDuration="1.540099213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:09.537222179 +0000 UTC m=+1427.592716007" watchObservedRunningTime="2026-10-30 12:56:09.540099213 +0000 UTC m=+1427.595593036"
Oct 30 12:56:10.036904 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:10.036875    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:10.038274 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:10.038251    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:56:10.038421 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:10.038337    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:56:11.037477 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:11.037447    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:56:11.037874 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:11.037501    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:56:12.036830 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:12.036797    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:56:12.037024 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:12.036849    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:56:12.395003 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:12.394923    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:56:13.037600 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:13.037567    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:56:13.037784 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:13.037613    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:56:14.036801 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:14.036768    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:56:14.037276 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:14.036815    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:56:15.037554 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:15.037519    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:56:15.037961 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:15.037571    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 12:56:16.674660 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:16.674615    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:16.675107 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:16.675053    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:16.675995 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:16.675962    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 12:56:25.263254 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.263219    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"]
Oct 30 12:56:25.267153 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.267135    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.300514 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.300490    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-bb86fd-openshift-default-dockercfg-bqgfs\""
Oct 30 12:56:25.317473 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.317450    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"]
Oct 30 12:56:25.436694 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.436660    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.436860 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.436714    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.436860 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.436778    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ef9e63-2828-4881-89e8-a00b74c5c68a-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.436860 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.436847    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.437014 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.436871    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.437074 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.437011    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.437131 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.437066    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.437186 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.437121    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.437243 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.437199    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvh9\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-kube-api-access-jkvh9\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538059 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.537986    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538059 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538018    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538059 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538052    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538074    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538097    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538126    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvh9\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-kube-api-access-jkvh9\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538182    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538250    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538580 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538433    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538580 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538535    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538695 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538597    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ef9e63-2828-4881-89e8-a00b74c5c68a-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538695 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538598    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.538801 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538766    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.539012 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.538995    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ef9e63-2828-4881-89e8-a00b74c5c68a-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.540980 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.540954    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.541166 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.541151    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.573803 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.573771    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.575154 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.575131    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvh9\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-kube-api-access-jkvh9\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.580009 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.579989    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:25.901409 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:56:25.901382    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ef9e63_2828_4881_89e8_a00b74c5c68a.slice/crio-5a7476bbabbbc4fbacfdec64323e882487a5a6b9e9bae41fdc63ae1f93040347 WatchSource:0}: Error finding container 5a7476bbabbbc4fbacfdec64323e882487a5a6b9e9bae41fdc63ae1f93040347: Status 404 returned error can't find the container with id 5a7476bbabbbc4fbacfdec64323e882487a5a6b9e9bae41fdc63ae1f93040347
Oct 30 12:56:25.903102 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.903001    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"]
Oct 30 12:56:25.903804 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.903777    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:25.903875 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.903838    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:25.903875 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:25.903867    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:26.579659 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:26.579614    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" event={"ID":"11ef9e63-2828-4881-89e8-a00b74c5c68a","Type":"ContainerStarted","Data":"dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336"}
Oct 30 12:56:26.580187 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:26.579676    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" event={"ID":"11ef9e63-2828-4881-89e8-a00b74c5c68a","Type":"ContainerStarted","Data":"5a7476bbabbbc4fbacfdec64323e882487a5a6b9e9bae41fdc63ae1f93040347"}
Oct 30 12:56:26.580339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:26.580318    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:26.582271 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:26.582242    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:56:26.582427 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:26.582299    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 12:56:26.675897 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:26.675833    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podStartSLOduration=1.675812519 podStartE2EDuration="1.675812519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:26.652835435 +0000 UTC m=+1444.708329260" watchObservedRunningTime="2026-10-30 12:56:26.675812519 +0000 UTC m=+1444.731306343"
Oct 30 12:56:27.580741 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:27.580707    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:56:27.581381 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:27.581340    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 12:56:28.581310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:28.580788    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:56:28.581310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:28.580868    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 12:56:28.937409 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:28.937369    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:29.131409 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:29.131365    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:29.378832 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:29.376402    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:29.587308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:29.586626    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:56:29.587308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:29.586688    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 12:56:29.958411 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:29.958367    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:29.963314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:29.959009    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:56:30.012318 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:30.009767    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:30.132407 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:30.132359    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:56:30.163311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:30.133339    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:56:30.441682 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:30.441612    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:56:30.448185 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:30.446756    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:56:30.581680 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:30.581645    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:56:30.581946 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:30.581918    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 12:56:31.003141 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.003111    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"]
Oct 30 12:56:31.007120 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.007101    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.027538 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.027516    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"]
Oct 30 12:56:31.028219 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.028195    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-69b689882f95\""
Oct 30 12:56:31.028333 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.028318    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vsw6g\""
Oct 30 12:56:31.074799 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.074754    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.074962 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.074830    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvbb\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-kube-api-access-nmvbb\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.074962 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.074872    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-plugins-config-volume\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.075111 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.074977    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-routing-overlay-volume\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.105828 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.105795    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 12:56:31.105948 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.105835    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 12:56:31.105948 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.105842    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 12:56:31.108802 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.108783    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 12:56:31.176415 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.176385    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvbb\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-kube-api-access-nmvbb\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.176551 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.176425    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-plugins-config-volume\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.176551 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.176514    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-routing-overlay-volume\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.176666 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.176644    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.177455 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.177403    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-routing-overlay-volume\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.177574 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.177426    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-plugins-config-volume\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.179336 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.179316    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.202300 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.202256    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvbb\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-kube-api-access-nmvbb\") pod \"payload-processing-external-model-c56dbf545-9hp6z\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") " pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.321699 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.321619    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:31.580864 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.580778    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:56:31.580864 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.580837    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 12:56:31.692581 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.692551    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"]
Oct 30 12:56:31.703335 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:31.703274    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z" event={"ID":"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16","Type":"ContainerStarted","Data":"174e58a3f03bd4506c867fd75b7a71fcbaad0fc47ac4fafac15bc479ed59a2d6"}
Oct 30 12:56:32.581055 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:32.581027    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 12:56:32.581449 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:32.581078    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 12:56:32.710982 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:32.710941    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z" event={"ID":"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16","Type":"ContainerStarted","Data":"0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a"}
Oct 30 12:56:32.711168 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:32.711008    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:32.787869 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:32.787816    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z" podStartSLOduration=2.787798621 podStartE2EDuration="2.787798621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:32.785403067 +0000 UTC m=+1450.840896891" watchObservedRunningTime="2026-10-30 12:56:32.787798621 +0000 UTC m=+1450.843292446"
Oct 30 12:56:32.811164 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:32.811140    4429 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-69b689882f95: secret "agc-cred-69b689882f95" not found
Oct 30 12:56:32.811164 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:32.811165    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z: secret "agc-cred-69b689882f95" not found
Oct 30 12:56:32.811390 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:32.811233    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials podName:2ad30ac2-01d6-44c5-b7e0-63c8a2913b16 nodeName:}" failed. No retries permitted until 2026-10-30 12:56:33.311216539 +0000 UTC m=+1451.366710360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials") pod "payload-processing-external-model-c56dbf545-9hp6z" (UID: "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16") : secret "agc-cred-69b689882f95" not found
Oct 30 12:56:33.209468 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:33.209444    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"]
Oct 30 12:56:33.318055 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:33.318024    4429 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-69b689882f95: secret "agc-cred-69b689882f95" not found
Oct 30 12:56:33.318055 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:33.318050    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z: secret "agc-cred-69b689882f95" not found
Oct 30 12:56:33.318321 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:33.318111    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials podName:2ad30ac2-01d6-44c5-b7e0-63c8a2913b16 nodeName:}" failed. No retries permitted until 2026-10-30 12:56:34.318092485 +0000 UTC m=+1452.373586304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials") pod "payload-processing-external-model-c56dbf545-9hp6z" (UID: "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16") : secret "agc-cred-69b689882f95" not found
Oct 30 12:56:33.585347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:33.585260    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:33.585843 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:33.585824    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:33.586590 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:33.586570    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 12:56:34.329519 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:34.329486    4429 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-69b689882f95: secret "agc-cred-69b689882f95" not found
Oct 30 12:56:34.329519 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:34.329509    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z: secret "agc-cred-69b689882f95" not found
Oct 30 12:56:34.329769 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:34.329565    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials podName:2ad30ac2-01d6-44c5-b7e0-63c8a2913b16 nodeName:}" failed. No retries permitted until 2026-10-30 12:56:36.329550058 +0000 UTC m=+1454.385043868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials") pod "payload-processing-external-model-c56dbf545-9hp6z" (UID: "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16") : secret "agc-cred-69b689882f95" not found
Oct 30 12:56:36.350099 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:36.350066    4429 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-69b689882f95: secret "agc-cred-69b689882f95" not found
Oct 30 12:56:36.350099 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:36.350092    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z: secret "agc-cred-69b689882f95" not found
Oct 30 12:56:36.350549 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:36.350154    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials podName:2ad30ac2-01d6-44c5-b7e0-63c8a2913b16 nodeName:}" failed. No retries permitted until 2026-10-30 12:56:40.350141844 +0000 UTC m=+1458.405635646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials") pod "payload-processing-external-model-c56dbf545-9hp6z" (UID: "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16") : secret "agc-cred-69b689882f95" not found
Oct 30 12:56:39.722764 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:39.722702    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z" podUID="2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" containerName="payload-processing" containerID="cri-o://0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a" gracePeriod=25
Oct 30 12:56:39.973937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:39.973878    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:40.143496 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.143463    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-plugins-config-volume\") pod \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") "
Oct 30 12:56:40.143496 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.143499    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-routing-overlay-volume\") pod \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") "
Oct 30 12:56:40.143722 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.143545    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-kube-api-access-nmvbb\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-kube-api-access-nmvbb\") pod \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") "
Oct 30 12:56:40.143722 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.143646    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials\") pod \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\" (UID: \"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16\") "
Oct 30 12:56:40.143886 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.143849    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-plugins-config-volume" pod "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" (UID: "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:40.143976 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.143893    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-routing-overlay-volume" pod "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" (UID: "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:56:40.144162 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.144128    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:56:40.144162 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.144159    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:56:40.146113 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.146092    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-kube-api-access-nmvbb" pod "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" (UID: "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16"). InnerVolumeSpecName "kube-api-access-nmvbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:40.146113 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.146108    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials" pod "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" (UID: "2ad30ac2-01d6-44c5-b7e0-63c8a2913b16"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:56:40.245164 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.245109    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:56:40.245164 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.245133    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmvbb\" (UniqueName: \"kubernetes.io/projected/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16-kube-api-access-nmvbb\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 12:56:40.813511 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.813480    4429 generic.go:356] "Generic (PLEG): container finished" podID="2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" containerID="0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a" exitCode=0
Oct 30 12:56:40.813987 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.813527    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"
Oct 30 12:56:40.813987 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.813571    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z" event={"ID":"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16","Type":"ContainerDied","Data":"0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a"}
Oct 30 12:56:40.813987 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.813639    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z" event={"ID":"2ad30ac2-01d6-44c5-b7e0-63c8a2913b16","Type":"ContainerDied","Data":"174e58a3f03bd4506c867fd75b7a71fcbaad0fc47ac4fafac15bc479ed59a2d6"}
Oct 30 12:56:40.813987 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.813672    4429 scope.go:122] "RemoveContainer" containerID="0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a"
Oct 30 12:56:40.824103 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.824084    4429 scope.go:122] "RemoveContainer" containerID="0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a"
Oct 30 12:56:40.824441 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:40.824420    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a\": container with ID starting with 0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a not found: ID does not exist" containerID="0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a"
Oct 30 12:56:40.824528 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.824453    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a"} err="failed to get container status \"0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a\": rpc error: code = NotFound desc = could not find container \"0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a\": container with ID starting with 0ad373ccb727fc11c65f2b5c29f611fc88402a9d50c44a460a6703f27895604a not found: ID does not exist"
Oct 30 12:56:40.869945 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.869919    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"]
Oct 30 12:56:40.883699 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:40.883674    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56dbf545-9hp6z"]
Oct 30 12:56:42.074525 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:56:42.074487    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" path="/var/lib/kubelet/pods/2ad30ac2-01d6-44c5-b7e0-63c8a2913b16/volumes"
Oct 30 12:56:42.371623 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:56:42.371540    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:57:12.370926 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:57:12.370840    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:57:12.678474 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.678387    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"]
Oct 30 12:57:12.678877 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.678856    4429 cpu_manager.go:414] "containerMap: removing container" podUID="2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" containerName="payload-processing"
Oct 30 12:57:12.678877 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.678874    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" containerName="payload-processing"
Oct 30 12:57:12.679074 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.679041    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad30ac2-01d6-44c5-b7e0-63c8a2913b16" containerName="payload-processing"
Oct 30 12:57:12.682409 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.682387    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.713776 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.713750    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e652ca\"/\"distinct-2-w4-e652ca-kserve-self-signed-certs\""
Oct 30 12:57:12.717084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.717056    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"]
Oct 30 12:57:12.877191 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.877146    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-model-cache\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.877410 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.877347    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tls-certs\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.877481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.877466    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tmp-dir\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.877570 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.877551    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-dshm\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.877657 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.877624    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-home\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.877814 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.877666    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8gq\" (UniqueName: \"kubernetes.io/projected/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-kube-api-access-7s8gq\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.978561 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.978532    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-dshm\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.978754 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.978588    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-home\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.978754 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.978609    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8gq\" (UniqueName: \"kubernetes.io/projected/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-kube-api-access-7s8gq\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.978754 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.978631    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-model-cache\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.978754 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.978669    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tls-certs\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.979043 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.978755    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tmp-dir\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.979155 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.979132    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-home\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.979224 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.979177    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-model-cache\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.979224 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.979200    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tmp-dir\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.981049 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.981007    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-dshm\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:12.981258 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:12.981240    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tls-certs\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:13.011768 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:13.011741    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8gq\" (UniqueName: \"kubernetes.io/projected/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-kube-api-access-7s8gq\") pod \"distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") " pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:13.295558 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:13.295480    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:13.601681 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:13.601646    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"]
Oct 30 12:57:13.602019 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:57:13.601717    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d4a09b0_d45a_4c6b_9c22_80c35693bc8e.slice/crio-bc69fcb883ff3c39aa4c8e86cf89e27b5e19a3e557158014834cc51bb641bdf0 WatchSource:0}: Error finding container bc69fcb883ff3c39aa4c8e86cf89e27b5e19a3e557158014834cc51bb641bdf0: Status 404 returned error can't find the container with id bc69fcb883ff3c39aa4c8e86cf89e27b5e19a3e557158014834cc51bb641bdf0
Oct 30 12:57:14.047864 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:14.047827    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" event={"ID":"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e","Type":"ContainerStarted","Data":"8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b"}
Oct 30 12:57:14.048038 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:14.047870    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" event={"ID":"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e","Type":"ContainerStarted","Data":"bc69fcb883ff3c39aa4c8e86cf89e27b5e19a3e557158014834cc51bb641bdf0"}
Oct 30 12:57:14.094042 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:14.093995    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podStartSLOduration=2.093982126 podStartE2EDuration="2.093982126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:14.092044079 +0000 UTC m=+1492.147537902" watchObservedRunningTime="2026-10-30 12:57:14.093982126 +0000 UTC m=+1492.149475976"
Oct 30 12:57:14.296024 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:14.295992    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:23.304310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:23.303499    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:23.361993 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:23.361959    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:24.307255 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:24.307222    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:57:37.291187 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.291154    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"]
Oct 30 12:57:37.296273 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.296254    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.347039 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.347016    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"]
Oct 30 12:57:37.363949 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.363924    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.364078 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.363978    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.364078 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.363997    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.364161 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.364108    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2a529-057f-4803-964d-1cb779d28da0-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.364161 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.364136    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2a529-057f-4803-964d-1cb779d28da0-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.364253 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.364211    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.364323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.364258    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.364370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.364340    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57k4\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-kube-api-access-x57k4\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.364370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.364362    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.465857 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.465829    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.465857 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.465858    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466069 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.465877    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2a529-057f-4803-964d-1cb779d28da0-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466069 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.465999    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2a529-057f-4803-964d-1cb779d28da0-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466181 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.466084    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466181 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.466163    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466321 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.466199    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57k4\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-kube-api-access-x57k4\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466321 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.466228    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466321 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.466234    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.466369    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466520 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.466477    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.466811 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.466743    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2a529-057f-4803-964d-1cb779d28da0-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.467021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.467004    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.467021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.467009    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.468316 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.468256    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.468741 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.468716    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2a529-057f-4803-964d-1cb779d28da0-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.497921 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.497900    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57k4\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-kube-api-access-x57k4\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.502977 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.502957    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.609949 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.609895    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:37.929521 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.929494    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"]
Oct 30 12:57:37.929837 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:57:37.929810    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa2a529_057f_4803_964d_1cb779d28da0.slice/crio-5bcf61c022c61636ae38265549d8bcc334d0e5522a20b0efb55ab842309dbe5c WatchSource:0}: Error finding container 5bcf61c022c61636ae38265549d8bcc334d0e5522a20b0efb55ab842309dbe5c: Status 404 returned error can't find the container with id 5bcf61c022c61636ae38265549d8bcc334d0e5522a20b0efb55ab842309dbe5c
Oct 30 12:57:37.932193 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.932162    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:37.932318 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.932225    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:37.932318 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:37.932256    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:38.385127 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:38.385089    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" event={"ID":"7aa2a529-057f-4803-964d-1cb779d28da0","Type":"ContainerStarted","Data":"18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421"}
Oct 30 12:57:38.385577 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:38.385145    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" event={"ID":"7aa2a529-057f-4803-964d-1cb779d28da0","Type":"ContainerStarted","Data":"5bcf61c022c61636ae38265549d8bcc334d0e5522a20b0efb55ab842309dbe5c"}
Oct 30 12:57:38.610209 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:38.610180    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:38.611514 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:38.611483    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:57:38.611654 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:38.611534    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 12:57:39.610917 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:39.610881    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:57:39.611343 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:39.610941    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 12:57:40.610388 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:40.610332    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:57:40.610578 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:40.610402    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 12:57:41.610699 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:41.610666    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:57:41.611084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:41.610723    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 12:57:42.395662 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:57:42.395626    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:57:42.610222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:42.610191    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:57:42.610432 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:42.610246    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 12:57:43.611169 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:43.611120    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:57:43.611668 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:43.611190    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 12:57:44.610461 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:44.610430    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 12:57:44.610626 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:44.610488    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 12:57:45.614961 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:45.614932    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:45.615384 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:45.615321    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:45.616120 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:45.616103    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 12:57:45.681400 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:45.681343    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podStartSLOduration=8.681328973 podStartE2EDuration="8.681328973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:38.457452339 +0000 UTC m=+1516.512946192" watchObservedRunningTime="2026-10-30 12:57:45.681328973 +0000 UTC m=+1523.736822797"
Oct 30 12:57:52.619972 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.619939    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"]
Oct 30 12:57:52.633215 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.633188    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.650389 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.650346    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"]
Oct 30 12:57:52.659681 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.659655    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ff0e6\"/\"distinct-w1-6ff0e6-kserve-self-signed-certs\""
Oct 30 12:57:52.749544 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.749508    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-home\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.749730 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.749555    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ftx\" (UniqueName: \"kubernetes.io/projected/2c0507ec-f598-4e58-ad65-2a0afb295eaf-kube-api-access-z2ftx\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.749730 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.749667    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-model-cache\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.749730 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.749703    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tmp-dir\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.749861 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.749822    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tls-certs\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.749962 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.749948    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-dshm\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.851107 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.851067    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-dshm\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.851364 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.851138    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-home\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.851364 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.851184    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ftx\" (UniqueName: \"kubernetes.io/projected/2c0507ec-f598-4e58-ad65-2a0afb295eaf-kube-api-access-z2ftx\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.851364 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.851234    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-model-cache\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.851364 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.851263    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tmp-dir\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.851364 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.851353    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tls-certs\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.851682 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.851621    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-home\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.851838 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.851805    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-model-cache\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.852053 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.852035    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tmp-dir\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.854053 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.854016    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-dshm\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.854584 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.854558    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tls-certs\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.876726 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.876654    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ftx\" (UniqueName: \"kubernetes.io/projected/2c0507ec-f598-4e58-ad65-2a0afb295eaf-kube-api-access-z2ftx\") pod \"distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") " pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:52.947039 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:52.947008    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:57:53.256058 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:53.256033    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"]
Oct 30 12:57:53.256254 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:57:53.256227    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c0507ec_f598_4e58_ad65_2a0afb295eaf.slice/crio-7bcbe55cec4c3d80d27c1431162b6eb59a9fba02e6ebda53958641b8ab90166e WatchSource:0}: Error finding container 7bcbe55cec4c3d80d27c1431162b6eb59a9fba02e6ebda53958641b8ab90166e: Status 404 returned error can't find the container with id 7bcbe55cec4c3d80d27c1431162b6eb59a9fba02e6ebda53958641b8ab90166e
Oct 30 12:57:53.477247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:53.477214    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" event={"ID":"2c0507ec-f598-4e58-ad65-2a0afb295eaf","Type":"ContainerStarted","Data":"907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e"}
Oct 30 12:57:53.477448 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:53.477258    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" event={"ID":"2c0507ec-f598-4e58-ad65-2a0afb295eaf","Type":"ContainerStarted","Data":"7bcbe55cec4c3d80d27c1431162b6eb59a9fba02e6ebda53958641b8ab90166e"}
Oct 30 12:57:53.532442 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:53.532310    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podStartSLOduration=1.5322669530000002 podStartE2EDuration="1.532266953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:53.530600597 +0000 UTC m=+1531.586094421" watchObservedRunningTime="2026-10-30 12:57:53.532266953 +0000 UTC m=+1531.587760778"
Oct 30 12:57:53.947691 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:57:53.947659    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:58:00.453312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:00.452384    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:00.453312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:00.452797    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:00.716380 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:00.708439    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qvm42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:00.716380 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:00.708524    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qvm42" podUID="c513e96c-4db5-43e5-bbd1-eda409197969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:00.748718 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:00.748682    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:00.749083 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:00.749053    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:01.317264 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.317220    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:01.464427 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.464377    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:01.465481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.465458    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:58:01.466342 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.464554    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:01.466879 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.466859    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:58:01.876459 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.876396    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-hxkhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:01.876809 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.876774    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:01.900413 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.900376    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-sjm4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:01.900673 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:01.900644    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:02.022991 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:02.022959    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:58:02.209912 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:02.209608    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:58:02.947201 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:02.947172    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:58:02.960844 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:02.960819    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:58:03.799157 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:03.799134    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:58:12.371660 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:58:12.371625    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:58:21.329111 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.329076    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"]
Oct 30 12:58:21.339664 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.339637    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.359872 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.359834    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"]
Oct 30 12:58:21.453603 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.453568    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4wc\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-kube-api-access-vj4wc\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.453788 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.453642    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.453788 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.453738    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.453788 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.453755    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.453891 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.453838    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.453924 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.453889    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.453963 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.453929    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.453997 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.453982    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.454092 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.454080    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.554723 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.554693    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.554929 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.554747    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.554929 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.554797    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4wc\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-kube-api-access-vj4wc\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.554929 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.554850    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.554929 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.554903    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.554929 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.554924    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.555176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.555004    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.555176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.555036    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.555176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.555097    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.555176 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.555148    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.555347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.555220    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.555521 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.555502    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.555625 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.555583    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.555978 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.555925    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.557578 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.557553    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.557890 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.557871    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.581673 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.581605    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4wc\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-kube-api-access-vj4wc\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.581987 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.581965    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:21.654431 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:21.654403    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:22.007583 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.007553    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"]
Oct 30 12:58:22.011728 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:58:22.011703    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4812a4_0f77_4113_855f_f0adb8ad1ea2.slice/crio-c3076c199420c9e6f63136acb936bed7d643bd6f03754c7d77b1795b3f6b752f WatchSource:0}: Error finding container c3076c199420c9e6f63136acb936bed7d643bd6f03754c7d77b1795b3f6b752f: Status 404 returned error can't find the container with id c3076c199420c9e6f63136acb936bed7d643bd6f03754c7d77b1795b3f6b752f
Oct 30 12:58:22.014612 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.014579    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:22.014684 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.014650    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:22.014684 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.014679    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:22.435058 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.435031    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"]
Oct 30 12:58:22.439372 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.439326    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.464734    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.464811    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.464874    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.464944    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.464980    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.465015    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.465063    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.465133    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3ca494-97b9-4fdc-8a51-672df9c78cda-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.469022 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.465185    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5zf\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-kube-api-access-gg5zf\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.507530 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.507483    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"]
Oct 30 12:58:22.511529 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.511494    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"]
Oct 30 12:58:22.521997 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.521969    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.524084 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.524054    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"]
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571075    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvzw\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-kube-api-access-zhvzw\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571138    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571188    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571235    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571266    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3ca494-97b9-4fdc-8a51-672df9c78cda-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571337    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5zf\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-kube-api-access-gg5zf\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571410    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571442    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571471    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571529    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571567    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571634    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571660    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571697    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571740    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5335b07-5214-489c-a4b2-ba57cdfd1043-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571801    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571839    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.574391 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.571879    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.587310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.577573    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.587310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.578524    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.587310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.579484    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3ca494-97b9-4fdc-8a51-672df9c78cda-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.587310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.579924    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.587310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.581338    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.587982 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.587957    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.592233 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.589803    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.604483 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.604454    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5zf\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-kube-api-access-gg5zf\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.604896 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.604869    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675020    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675078    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5335b07-5214-489c-a4b2-ba57cdfd1043-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675166    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvzw\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-kube-api-access-zhvzw\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675214    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675257    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675370    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675407    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675459    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.675528    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.676997    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5335b07-5214-489c-a4b2-ba57cdfd1043-istiod-ca-cert\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.677277    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.678347 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.678129    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-data\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.679000 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.678610    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-credential-socket\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.679000 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.678675    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-certs\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.680377 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.680353    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-envoy\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.680637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.680620    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-podinfo\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.702734 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.702669    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-token\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.704725 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.704708    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvzw\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-kube-api-access-zhvzw\") pod \"e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") " pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.769309 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.768537    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:22.842845 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.842816    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:22.909956 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.909911    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" event={"ID":"9b4812a4-0f77-4113-855f-f0adb8ad1ea2","Type":"ContainerStarted","Data":"07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596"}
Oct 30 12:58:22.910164 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.909972    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" event={"ID":"9b4812a4-0f77-4113-855f-f0adb8ad1ea2","Type":"ContainerStarted","Data":"c3076c199420c9e6f63136acb936bed7d643bd6f03754c7d77b1795b3f6b752f"}
Oct 30 12:58:22.982868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:22.981351    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podStartSLOduration=1.981328538 podStartE2EDuration="1.981328538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:22.959400569 +0000 UTC m=+1561.014894414" watchObservedRunningTime="2026-10-30 12:58:22.981328538 +0000 UTC m=+1561.036822363"
Oct 30 12:58:23.086514 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:58:23.086362    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3ca494_97b9_4fdc_8a51_672df9c78cda.slice/crio-cd3ef01b0a4016729966a1de83e51a04b3f2ce5c4415e78810fe832ada1fe1ee WatchSource:0}: Error finding container cd3ef01b0a4016729966a1de83e51a04b3f2ce5c4415e78810fe832ada1fe1ee: Status 404 returned error can't find the container with id cd3ef01b0a4016729966a1de83e51a04b3f2ce5c4415e78810fe832ada1fe1ee
Oct 30 12:58:23.087039 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.087012    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"]
Oct 30 12:58:23.089083 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.089053    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:23.089154 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.089142    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:23.089196 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.089184    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:23.188686 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:58:23.188657    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5335b07_5214_489c_a4b2_ba57cdfd1043.slice/crio-b1c320abd53f41bfcab6934e9b3301252383cc03b711fef8e7081b00025577d5 WatchSource:0}: Error finding container b1c320abd53f41bfcab6934e9b3301252383cc03b711fef8e7081b00025577d5: Status 404 returned error can't find the container with id b1c320abd53f41bfcab6934e9b3301252383cc03b711fef8e7081b00025577d5
Oct 30 12:58:23.189786 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.189763    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"]
Oct 30 12:58:23.191825 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.191793    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:23.191918 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.191877    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:23.191960 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.191923    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:23.667371 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.664646    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:23.681414 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.669420    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 12:58:23.681414 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.669486    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 12:58:23.845128 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.845082    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"]
Oct 30 12:58:23.859324 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.856672    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.876310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.872879    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"]
Oct 30 12:58:23.898354 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.895447    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:23.898354 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.895905    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"]
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.925725    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"]
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928569    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928672    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928710    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928765    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928787    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928809    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928832    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mng\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-kube-api-access-p5mng\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928898    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.931361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.928982    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:23.937499 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.936514    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" event={"ID":"c5335b07-5214-489c-a4b2-ba57cdfd1043","Type":"ContainerStarted","Data":"94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f"}
Oct 30 12:58:23.937499 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.936583    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" event={"ID":"c5335b07-5214-489c-a4b2-ba57cdfd1043","Type":"ContainerStarted","Data":"b1c320abd53f41bfcab6934e9b3301252383cc03b711fef8e7081b00025577d5"}
Oct 30 12:58:23.945310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.940399    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" event={"ID":"af3ca494-97b9-4fdc-8a51-672df9c78cda","Type":"ContainerStarted","Data":"ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e"}
Oct 30 12:58:23.945310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:23.940450    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" event={"ID":"af3ca494-97b9-4fdc-8a51-672df9c78cda","Type":"ContainerStarted","Data":"cd3ef01b0a4016729966a1de83e51a04b3f2ce5c4415e78810fe832ada1fe1ee"}
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030073    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030174    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030231    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030272    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030370    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030397    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030423    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030449    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mng\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-kube-api-access-p5mng\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030478    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030582    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030617    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030650    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030685    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030784    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9hfk\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-kube-api-access-r9hfk\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030828    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030910    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.030968    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.031048    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.032243    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.033075    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.035472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.034065    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.040525 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.037124    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.040525 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.037761    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.046910 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.045644    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.079503    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podStartSLOduration=2.07948089 podStartE2EDuration="2.07948089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:24.076828478 +0000 UTC m=+1562.132322305" watchObservedRunningTime="2026-10-30 12:58:24.07948089 +0000 UTC m=+1562.134974714"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.095171    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.132749    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.132867    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.132900    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.132934    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.132994    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hfk\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-kube-api-access-r9hfk\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.133068    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.133106    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.133217    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.133362    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.133820    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.134058    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.134768    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.138351 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.137731    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.139649 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.139595    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.141591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.141564    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.143658 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.143601    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podStartSLOduration=2.143583794 podStartE2EDuration="2.143583794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:24.131037301 +0000 UTC m=+1562.186531126" watchObservedRunningTime="2026-10-30 12:58:24.143583794 +0000 UTC m=+1562.199077620"
Oct 30 12:58:24.144698 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.144682    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.144963 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.144918    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.145880 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.145864    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mng\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-kube-api-access-p5mng\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.182369 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.180224    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.231141 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.230809    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:24.231141 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.230986    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hfk\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-kube-api-access-r9hfk\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.246074 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.245857    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:24.679313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.655702    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 12:58:24.679313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.655779    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 12:58:24.739584 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.739550    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"]
Oct 30 12:58:24.745460 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.745421    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:24.745592 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.745537    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:24.745669 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.745589    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:24.835360 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.832586    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:24.835360 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.834413    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 12:58:24.835360 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.834466    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 12:58:24.844686 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.844663    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:24.844932 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.844911    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 12:58:24.845067 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:24.845046    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 12:58:25.027370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.026870    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" event={"ID":"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5","Type":"ContainerStarted","Data":"bf22f35c9d9e3759b458660ee22a2dc1381c7db99e9e53065766de7b3894b3b8"}
Oct 30 12:58:25.027370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.027154    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"]
Oct 30 12:58:25.078881 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:58:25.078018    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf11c9b_c168_4347_bd9b_c1c3aa757027.slice/crio-30c83567f612eb79e71d4882b7255ccd9654df3947043b76a6840956254eba0a WatchSource:0}: Error finding container 30c83567f612eb79e71d4882b7255ccd9654df3947043b76a6840956254eba0a: Status 404 returned error can't find the container with id 30c83567f612eb79e71d4882b7255ccd9654df3947043b76a6840956254eba0a
Oct 30 12:58:25.131566 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.130155    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:25.131566 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.130257    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:25.131566 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.130328    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:25.668419 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.667964    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 12:58:25.668419 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.668036    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 12:58:25.789712 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.789663    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 12:58:25.790334 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.790269    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 12:58:25.848200 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.848168    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 12:58:25.848472 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:25.848437    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 12:58:26.055324 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.055267    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" event={"ID":"aaf11c9b-c168-4347-bd9b-c1c3aa757027","Type":"ContainerStarted","Data":"f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93"}
Oct 30 12:58:26.055624 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.055599    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" event={"ID":"aaf11c9b-c168-4347-bd9b-c1c3aa757027","Type":"ContainerStarted","Data":"30c83567f612eb79e71d4882b7255ccd9654df3947043b76a6840956254eba0a"}
Oct 30 12:58:26.069834 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.069795    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" event={"ID":"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5","Type":"ContainerStarted","Data":"2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df"}
Oct 30 12:58:26.240888 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.240853    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:26.241380 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.241351    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:26.255527 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.255496    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:26.255767 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.255740    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:26.292310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.281801    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:26.292310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.281874    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:26.307481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.307371    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podStartSLOduration=3.307352222 podStartE2EDuration="3.307352222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:26.277030429 +0000 UTC m=+1564.332524258" watchObservedRunningTime="2026-10-30 12:58:26.307352222 +0000 UTC m=+1564.362846066"
Oct 30 12:58:26.658369 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.655716    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 12:58:26.658369 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.655786    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 12:58:26.770311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.769758    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 12:58:26.770311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.769824    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 12:58:26.844020 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.843974    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 12:58:26.845115 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:26.845078    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 12:58:27.250311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.249984    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:27.250311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.250045    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:27.254311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.250995    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:27.254311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.251046    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:27.673783 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.673228    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 12:58:27.673783 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.673308    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 12:58:27.780428 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.771667    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 12:58:27.780428 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.771729    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 12:58:27.848321 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.845947    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 12:58:27.848321 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.846007    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 12:58:27.998329 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:27.996833    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:28.027582 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.027542    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:28.213311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.212377    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:28.238310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.235937    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:28.238310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.236000    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:28.273370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.264599    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:28.273370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.264655    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:28.659370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.658696    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 12:58:28.659370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.658755    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 12:58:28.773182 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.772709    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 12:58:28.773182 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.772778    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 12:58:28.801311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.791866    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:28.844185 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.843713    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 12:58:28.844185 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.843780    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 12:58:28.997306 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.997243    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:28.998590 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:28.998568    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:58:29.031689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.031651    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:29.032529 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.032506    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:58:29.063902 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.063836    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podStartSLOduration=6.063815003 podStartE2EDuration="6.063815003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:26.339646606 +0000 UTC m=+1564.395140432" watchObservedRunningTime="2026-10-30 12:58:29.063815003 +0000 UTC m=+1567.119308829"
Oct 30 12:58:29.243371 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.236661    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 12:58:29.525656 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.525615    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:58:29.529250 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.247449    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:29.529250 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.526512    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:29.529250 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.491473    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm" podUID="3fbb3787-0ead-4228-bc7a-a8b85cddce02" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:29.529250 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.310365    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:29.560310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.398449    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:29.560310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.288782    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:29.560310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.554490    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:29.723901 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.723856    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 12:58:29.724101 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.723930    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 12:58:29.773826 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.773784    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 12:58:29.774109 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.774079    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 12:58:29.826367 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.806511    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:29.826367 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.806948    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:58:29.957585 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.957545    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 12:58:29.957881 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:29.957852    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 12:58:30.049049 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.048997    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:30.055498 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.055447    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 12:58:30.251636 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.251602    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:30.251880 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.251856    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:30.295352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.252217    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:30.295352 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.289382    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:30.530010 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.529908    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 12:58:30.530648 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.530627    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:58:30.534297 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.533730    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:30.536035 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.536018    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:58:30.731312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.729851    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:58:30.811691 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.811587    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:30.835349 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.814208    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 12:58:30.860362 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.860317    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 12:58:31.004313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.913804    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 12:58:31.004313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:30.962697    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 12:58:31.054673 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.054458    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:31.248313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.234755    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:31.248313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.236981    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:31.248313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.244447    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 12:58:31.270007 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.269971    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:31.270388 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.270357    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:31.285609 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.285575    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:31.289421 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.289358    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:31.530804 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.530709    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 12:58:31.574579 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.573888    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:58:31.743197 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.743167    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:58:31.745480 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.745385    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:58:31.782364 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.781760    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 12:58:31.782364 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.781876    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 12:58:31.848242 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.848199    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 12:58:31.852727 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:31.852519    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 12:58:32.271325 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:32.235858    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:32.271325 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:32.235921    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:32.303320 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:32.293140    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:32.303320 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:32.293217    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:33.232385 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.231662    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:33.232385 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.231722    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:33.264346 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.253685    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:33.264346 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.253743    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:33.754314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.738720    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:33.754314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.749053    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:33.795312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.792355    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:33.795312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.792469    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:33.827490 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.821025    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:33.898307 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.897586    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:33.898307 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.897674    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:33.963307 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:33.958441    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:34.084368 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.073389    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:34.232727 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.232681    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:34.232926 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.232761    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:34.296481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.296429    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:34.296753 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.296724    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:34.298486 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.298451    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded"
Oct 30 12:58:34.538432 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.538381    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:34.764851 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.764809    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:34.765846 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.765824    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:58:34.786353 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.786308    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:34.786923 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.786904    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:58:34.860368 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.848005    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:34.860368 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.856373    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:58:34.860743 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.860717    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:34.860950 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.860925    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:34.958452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.958413    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:34.958711 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:34.958675    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:35.076313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.075589    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:35.076313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.076062    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:58:35.334702 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.334663    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 12:58:35.348316 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.343305    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 12:58:35.348583 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.334983    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:35.348718 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.348693    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:35.540432 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.540382    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:35.591911 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.591821    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:58:35.770312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.769419    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 12:58:35.824308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.791490    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:35.867629 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:35.867515    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 12:58:36.017484 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.017440    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:36.080240 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.080204    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:36.081850 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.079359    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:36.096867 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.096840    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:36.097045 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.097018    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:36.197442 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.184973    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:58:36.223342 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.207182    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:58:36.256310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.251203    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 12:58:36.256310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.251274    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 12:58:36.323316 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.323034    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:58:36.453724 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.453645    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:36.454906 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.454885    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:36.478693 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.478630    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 12:58:36.481395 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.480537    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:58:36.498796 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.498778    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:58:36.591109 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.590710    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:36.591989 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.591827    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:36.597409 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.597390    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 12:58:36.834784 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.834695    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:36.835454 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.835434    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:36.836572 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:36.836555    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 12:58:38.287444 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:38.287397    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:58:38.287884 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:38.287457    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:38.476172 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:38.476146    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:38.476581 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:38.476553    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:38.477708 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:38.477692    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 12:58:40.706209 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.706176    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"]
Oct 30 12:58:40.711456 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.711436    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.737406 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.737381    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"]
Oct 30 12:58:40.778937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.778901    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.779180 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.778962    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.779180 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.779067    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.779180 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.779103    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.779180 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.779159    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.779563 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.779192    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b788p\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-kube-api-access-b788p\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.779563 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.779232    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.779563 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.779311    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.779798 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.779566    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880480 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880454    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880592 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880485    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880592 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880512    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880592 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880533    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b788p\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-kube-api-access-b788p\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880592 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880556    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880592 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880575    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880644    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880719    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.880868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880754    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.881026 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.880987    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.881083 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.881059    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.881193 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.881174    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.881267 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.881210    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.881415 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.881393    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.883741 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.883714    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.883953 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.883934    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.905772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.905751    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:40.906002 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:40.905985    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b788p\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-kube-api-access-b788p\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:41.026247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:41.026177    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:41.340001 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:58:41.339967    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5795dc_ef5d_4eaf_8505_b2ad430adfcd.slice/crio-6be9963f5df29ccab72235a0955e240e3e34c736003ece452a9ec7e84de5d14d WatchSource:0}: Error finding container 6be9963f5df29ccab72235a0955e240e3e34c736003ece452a9ec7e84de5d14d: Status 404 returned error can't find the container with id 6be9963f5df29ccab72235a0955e240e3e34c736003ece452a9ec7e84de5d14d
Oct 30 12:58:41.341190 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:41.341168    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"]
Oct 30 12:58:41.342375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:41.342347    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:41.342462 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:41.342407    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:41.342462 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:41.342438    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:42.262876 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:42.262830    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" event={"ID":"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd","Type":"ContainerStarted","Data":"f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92"}
Oct 30 12:58:42.263403 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:42.262894    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" event={"ID":"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd","Type":"ContainerStarted","Data":"6be9963f5df29ccab72235a0955e240e3e34c736003ece452a9ec7e84de5d14d"}
Oct 30 12:58:42.334582 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:42.334531    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podStartSLOduration=2.334516112 podStartE2EDuration="2.334516112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:42.331624683 +0000 UTC m=+1580.387118526" watchObservedRunningTime="2026-10-30 12:58:42.334516112 +0000 UTC m=+1580.390009936"
Oct 30 12:58:42.398017 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:58:42.397984    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:58:43.027350 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:43.027319    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:43.029188 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:43.029164    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:58:43.029331 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:43.029211    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 12:58:44.027478 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:44.027446    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:58:44.027865 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:44.027495    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 12:58:45.027037 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:45.026998    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:58:45.027211 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:45.027053    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 12:58:46.026603 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:46.026569    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:58:46.027020 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:46.026644    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 12:58:47.026961 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:47.026928    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:58:47.027373 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:47.026978    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 12:58:48.058314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:48.047596    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:58:48.058314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:48.047664    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 12:58:48.890433 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:48.890382    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:48.891054 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:48.891024    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.026664 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:49.026629    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 12:58:49.026937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:49.026908    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 12:58:49.110425 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:49.110373    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.111389 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:49.111357    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.214310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:49.213373    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.231481 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:49.231434    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-549nm" podUID="3fbb3787-0ead-4228-bc7a-a8b85cddce02" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:49.369432 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:49.369384    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:58:50.031743 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:50.031710    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:50.032097 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:50.032077    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:50.032769 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:50.032748    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 12:58:51.469114 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.469083    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"]
Oct 30 12:58:51.474222 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.474196    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.492266 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.492240    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.492541 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.492519    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27tht\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-kube-api-access-27tht\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.492647 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.492566    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.492770 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.492750    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.492830 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.492797    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.492887 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.492841    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.492944 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.492910    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.493006 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.492951    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.493269 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.493244    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.496866 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.496845    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"]
Oct 30 12:58:51.594311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594262    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27tht\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-kube-api-access-27tht\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594441 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594330    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594441 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594399    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594441 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594429    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594448    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594476    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594503    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594529    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594591 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594560    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594844 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594767    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594894 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594840    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-data\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.594953 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.594934    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-certs\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.595037 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.595019    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-credential-socket\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.595153 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.595126    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istiod-ca-cert\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.596921 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.596905    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-envoy\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.597065 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.597047    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-podinfo\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.620300 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.620266    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-token\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.620399 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.620350    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27tht\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-kube-api-access-27tht\") pod \"e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") " pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:51.790521 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:51.790438    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:52.109893 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:52.109857    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"]
Oct 30 12:58:52.115008 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:52.114973    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:52.115142 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:52.115037    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:52.115142 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:52.115066    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:52.425225 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:52.425144    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" event={"ID":"11715a29-3dfa-44d8-8eaa-61ecc1f56143","Type":"ContainerStarted","Data":"ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07"}
Oct 30 12:58:52.425225 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:52.425200    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" event={"ID":"11715a29-3dfa-44d8-8eaa-61ecc1f56143","Type":"ContainerStarted","Data":"e027862355941cfd16906f55339796a980755606c6dbede9e2d969b7881f9bf0"}
Oct 30 12:58:52.475109 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:52.475059    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" podStartSLOduration=1.475044688 podStartE2EDuration="1.475044688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:52.47409034 +0000 UTC m=+1590.529584164" watchObservedRunningTime="2026-10-30 12:58:52.475044688 +0000 UTC m=+1590.530538511"
Oct 30 12:58:52.790737 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:52.790708    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:53.792641 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:53.792602    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:58:53.793036 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:53.792651    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:58:54.791051 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:54.791019    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:58:54.793149 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:54.791075    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:58:55.791520 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:55.791480    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:58:55.791724 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:55.791557    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:58:56.791146 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:56.791101    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:58:56.791567 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:56.791167    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:58:57.791202 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:57.791162    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:58:57.791629 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:57.791237    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:58:58.791100 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:58.791060    4429 patch_prober.go:29] interesting pod/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 12:58:58.791313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:58.791125    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 12:58:59.795857 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:59.795825    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:59.796366 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:59.796163    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:58:59.796938 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:58:59.796911    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 12:59:05.780780 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:05.780729    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 12:59:05.873449 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:05.873393    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:06.085018 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.084918    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 12:59:06.114418 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.114378    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:06.233170 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.233125    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:06.343471 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.336728    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"]
Oct 30 12:59:06.345965 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.345936    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.365329 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.365281    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"]
Oct 30 12:59:06.384411 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.379345    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6ff0e6-openshift-default-dockercfg-xpjcg\""
Oct 30 12:59:06.560314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.524674    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.564465 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.564426    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.568462 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.568433    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvp2\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-kube-api-access-blvp2\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.572493 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.572465    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.628370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.626864    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.628370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.626932    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.628370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.627078    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.628370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.627168    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.628370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.627271    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738238    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738322    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738373    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738408    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvp2\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-kube-api-access-blvp2\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738474    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738517    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738543    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738657    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.738741    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.739239    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.739571    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.740231    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.740759    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.752312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.746146    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.827332 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.804350    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 12:59:06.827332 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.805077    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:59:06.827332 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.811792    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.875309 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.857586    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.933425 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.884982    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvp2\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-kube-api-access-blvp2\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:06.933425 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.885079    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:06.933425 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:06.885501    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:59:07.096560 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.096519    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.134886 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.097513    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:59:07.134886 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.123315    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.134886 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.123890    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:59:07.438313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.413373    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 12:59:07.440247 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.440220    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:59:07.474777 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.467981    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:59:07.475074 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.475041    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.650314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.646405    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:59:07.698689 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.672121    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.821310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.820643    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 12:59:07.893326 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.891493    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:59:07.893326 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.891556    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.893326 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.891639    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:07.958390 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:07.941524    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:08.135313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:08.099372    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:08.135313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:08.130399    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:08.258359 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:08.257644    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:08.445758 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:08.445721    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 12:59:08.895963 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:08.895910    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 12:59:08.932987 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:08.932943    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:08.934375 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:08.934327    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:09.174370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.126943    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 12:59:09.174370 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.136377    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:09.445937 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.445809    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:59:09.448320 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.447126    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 12:59:09.448736 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.447199    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:09.485890 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.485866    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:59:09.547478 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.547001    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:59:09.555318 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.552619    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:59:09.555318 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.554253    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:59:09.752173 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:59:09.752142    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcbb19a_e0e3_46d6_adcf_405ca61eb41b.slice/crio-db3c7c91cc6d28ff7e703ce717f0bf0c0b2d6dbbad2aa6ff14756b26a072d33b WatchSource:0}: Error finding container db3c7c91cc6d28ff7e703ce717f0bf0c0b2d6dbbad2aa6ff14756b26a072d33b: Status 404 returned error can't find the container with id db3c7c91cc6d28ff7e703ce717f0bf0c0b2d6dbbad2aa6ff14756b26a072d33b
Oct 30 12:59:09.752502 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.752475    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"]
Oct 30 12:59:09.755048 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.755013    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:09.755141 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.755093    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:09.755141 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:09.755133    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:10.228304 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:10.225539    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" event={"ID":"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b","Type":"ContainerStarted","Data":"f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd"}
Oct 30 12:59:10.228304 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:10.225604    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" event={"ID":"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b","Type":"ContainerStarted","Data":"db3c7c91cc6d28ff7e703ce717f0bf0c0b2d6dbbad2aa6ff14756b26a072d33b"}
Oct 30 12:59:10.269309 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:10.268655    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:10.273886 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:10.273857    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:59:10.274062 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:10.273904    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:59:11.272714 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:11.272680    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:59:11.273253 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:11.272738    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:59:12.271314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:12.270679    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:59:12.271314 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:12.270747    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:59:12.399356 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:59:12.399315    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:59:13.265768 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:13.265724    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:59:13.266083 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:13.266054    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:59:14.268319 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:14.267407    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:59:14.268319 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:14.267476    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:59:15.272518 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:15.272462    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:59:15.272946 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:15.272550    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:59:16.263558 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:16.263523    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 12:59:16.263744 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:16.263578    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 12:59:17.268717 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:17.268685    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:17.269202 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:17.269097    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:17.269854 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:17.269835    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 12:59:17.334672 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:17.334621    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podStartSLOduration=11.334604509 podStartE2EDuration="11.334604509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:10.285100783 +0000 UTC m=+1608.340594607" watchObservedRunningTime="2026-10-30 12:59:17.334604509 +0000 UTC m=+1615.390098332"
Oct 30 12:59:22.461310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:22.449578    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 12:59:22.947361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:22.936299    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.148808 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:23.148757    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 12:59:23.523334 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:23.499528    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 12:59:23.523334 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:23.513387    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.523334 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:23.513835    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:59:23.553745 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:23.552461    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.941671 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:23.941623    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:23.943673 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:23.943622    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:59:23.973312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:23.962382    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.215418 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.204701    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.259349 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.245478    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"]
Oct 30 12:59:24.323662 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.323619    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.368990 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.368958    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:59:24.370098 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.370077    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.505299 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.504064    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506226    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506330    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506383    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506415    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506440    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506496    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506629    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506699    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7v89\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-kube-api-access-t7v89\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506796    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.515310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.506974    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:59:24.540349 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.520790    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 12:59:24.616344 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.590959    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 12:59:24.900639 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.774575    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.900900 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.900875    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.901105 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.901088    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.901268 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.901251    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.981313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.950948    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.981313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.950995    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.981313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.951083    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.981313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.951246    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.999309 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.990462    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7v89\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-kube-api-access-t7v89\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:24.999309 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.911863    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.004308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.999593    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.004308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.912444    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:59:25.004308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.000000    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.004308 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.775024    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.021311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:24.942144    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:25.092691 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.092662    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.128193 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.128097    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.225310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.184798    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"]
Oct 30 12:59:25.225892 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.225849    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:25.405360 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.399445    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.492716 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.492680    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.507318 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.507235    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 12:59:25.534318 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.513069    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7v89\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-kube-api-access-t7v89\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.551342 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.550639    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:25.582451 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.576429    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:25.898469 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:25.898426    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:26.188369 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:26.187430    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 12:59:26.259306 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:26.242270    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 12:59:26.536690 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:26.528486    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:26.588460 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:26.588383    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:27.126324 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.106504    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:27.159809 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.159778    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:59:27.278057 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.278028    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"]
Oct 30 12:59:27.279640 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.215176    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:27.280000 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.279947    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:27.280262 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.280219    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:27.344612 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.335886    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:59:27.380032 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.380005    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:59:27.407898 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.407873    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:59:27.427501 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.427475    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:59:27.441385 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:27.441351    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" event={"ID":"f55cd737-74cc-4679-8dd5-fa41d82dbc47","Type":"ContainerStarted","Data":"5f3ead7d5534abe2e0dae9b81c29f4bc2bf42ef0a6fe9c832a447dfe265b446d"}
Oct 30 12:59:28.478310 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:28.478147    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" event={"ID":"f55cd737-74cc-4679-8dd5-fa41d82dbc47","Type":"ContainerStarted","Data":"c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3"}
Oct 30 12:59:28.624407 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:28.624364    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:28.635049 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:28.635022    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:59:28.635238 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:28.635072    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:59:28.644633 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:28.644570    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" podStartSLOduration=4.644548353 podStartE2EDuration="4.644548353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:28.568651438 +0000 UTC m=+1626.624145333" watchObservedRunningTime="2026-10-30 12:59:28.644548353 +0000 UTC m=+1626.700042177"
Oct 30 12:59:29.572313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:29.571563    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:59:29.572313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:29.571617    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:59:30.569646 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:30.569611    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:59:30.569975 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:30.569936    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:59:31.567312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:31.566699    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:59:31.567312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:31.566763    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:59:32.552687 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:32.552653    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:59:32.552968 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:32.552935    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:59:33.555498 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:33.555464    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 12:59:33.555957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:33.555522    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 12:59:34.509253 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.509219    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"]
Oct 30 12:59:34.513583 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.513561    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:34.699339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.663941    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-87cae1-dockercfg-nt24h\""
Oct 30 12:59:34.699339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.665091    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-87cae1\""
Oct 30 12:59:34.699339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.668860    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c3d26-4b6f-4b98-9901-81c197bc7815-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr\" (UID: \"976c3d26-4b6f-4b98-9901-81c197bc7815\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:34.699339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.697236    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxct\" (UniqueName: \"kubernetes.io/projected/976c3d26-4b6f-4b98-9901-81c197bc7815-kube-api-access-vpxct\") pod \"payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr\" (UID: \"976c3d26-4b6f-4b98-9901-81c197bc7815\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:34.699339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.675922    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"]
Oct 30 12:59:34.699339 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.697467    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"]
Oct 30 12:59:34.701570 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.701547    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"]
Oct 30 12:59:34.701813 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.701768    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:34.799184 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.799096    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c3d26-4b6f-4b98-9901-81c197bc7815-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr\" (UID: \"976c3d26-4b6f-4b98-9901-81c197bc7815\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:34.799378 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.799263    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxct\" (UniqueName: \"kubernetes.io/projected/976c3d26-4b6f-4b98-9901-81c197bc7815-kube-api-access-vpxct\") pod \"payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr\" (UID: \"976c3d26-4b6f-4b98-9901-81c197bc7815\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:34.799378 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.799353    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e511ae-0712-46f3-9088-2c763a5f2c11-plugins-config-volume\") pod \"payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst\" (UID: \"33e511ae-0712-46f3-9088-2c763a5f2c11\") " pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:34.799565 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.799544    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pprw\" (UniqueName: \"kubernetes.io/projected/33e511ae-0712-46f3-9088-2c763a5f2c11-kube-api-access-6pprw\") pod \"payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst\" (UID: \"33e511ae-0712-46f3-9088-2c763a5f2c11\") " pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:34.800638 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.800613    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c3d26-4b6f-4b98-9901-81c197bc7815-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr\" (UID: \"976c3d26-4b6f-4b98-9901-81c197bc7815\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:34.826004 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.825964    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxct\" (UniqueName: \"kubernetes.io/projected/976c3d26-4b6f-4b98-9901-81c197bc7815-kube-api-access-vpxct\") pod \"payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr\" (UID: \"976c3d26-4b6f-4b98-9901-81c197bc7815\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:34.829690 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.829658    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:34.907708 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.905924    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e511ae-0712-46f3-9088-2c763a5f2c11-plugins-config-volume\") pod \"payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst\" (UID: \"33e511ae-0712-46f3-9088-2c763a5f2c11\") " pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:34.907708 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.906098    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pprw\" (UniqueName: \"kubernetes.io/projected/33e511ae-0712-46f3-9088-2c763a5f2c11-kube-api-access-6pprw\") pod \"payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst\" (UID: \"33e511ae-0712-46f3-9088-2c763a5f2c11\") " pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:34.907708 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.907452    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e511ae-0712-46f3-9088-2c763a5f2c11-plugins-config-volume\") pod \"payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst\" (UID: \"33e511ae-0712-46f3-9088-2c763a5f2c11\") " pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:34.936323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:34.936244    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pprw\" (UniqueName: \"kubernetes.io/projected/33e511ae-0712-46f3-9088-2c763a5f2c11-kube-api-access-6pprw\") pod \"payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst\" (UID: \"33e511ae-0712-46f3-9088-2c763a5f2c11\") " pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:35.032683 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.032554    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:35.053807 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.053782    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:35.054957 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.054929    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:35.057538 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.057477    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 12:59:35.146895 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.146867    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"]
Oct 30 12:59:35.416945 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:59:35.416917    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e511ae_0712_46f3_9088_2c763a5f2c11.slice/crio-dd0280a3262bbb952cfed9bcce9b4ed8b8652113554cb51ff011560c9451c65f WatchSource:0}: Error finding container dd0280a3262bbb952cfed9bcce9b4ed8b8652113554cb51ff011560c9451c65f: Status 404 returned error can't find the container with id dd0280a3262bbb952cfed9bcce9b4ed8b8652113554cb51ff011560c9451c65f
Oct 30 12:59:35.417851 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.417827    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"]
Oct 30 12:59:35.538866 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.538827    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst" event={"ID":"33e511ae-0712-46f3-9088-2c763a5f2c11","Type":"ContainerStarted","Data":"a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b"}
Oct 30 12:59:35.539027 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.538926    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:35.539027 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.538953    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst" event={"ID":"33e511ae-0712-46f3-9088-2c763a5f2c11","Type":"ContainerStarted","Data":"dd0280a3262bbb952cfed9bcce9b4ed8b8652113554cb51ff011560c9451c65f"}
Oct 30 12:59:35.540556 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.540523    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr" event={"ID":"976c3d26-4b6f-4b98-9901-81c197bc7815","Type":"ContainerStarted","Data":"8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc"}
Oct 30 12:59:35.540687 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.540571    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr" event={"ID":"976c3d26-4b6f-4b98-9901-81c197bc7815","Type":"ContainerStarted","Data":"79d07451d1cb22ba5ad60d62f2a4a7ce10f72a93260258351cf6b458e493e029"}
Oct 30 12:59:35.587355 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.587275    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst" podStartSLOduration=1.587263292 podStartE2EDuration="1.587263292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:35.585201448 +0000 UTC m=+1633.640695272" watchObservedRunningTime="2026-10-30 12:59:35.587263292 +0000 UTC m=+1633.642757116"
Oct 30 12:59:35.631590 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:35.631548    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr" podStartSLOduration=1.631535618 podStartE2EDuration="1.631535618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:35.628142879 +0000 UTC m=+1633.683636704" watchObservedRunningTime="2026-10-30 12:59:35.631535618 +0000 UTC m=+1633.687029474"
Oct 30 12:59:36.547765 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:36.547732    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:38.594435 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.594404    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"]
Oct 30 12:59:38.599039 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.599013    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.630156 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.630128    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"]
Oct 30 12:59:38.630385 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.630364    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ea117f29-openshift-default-dockercfg-829tn\""
Oct 30 12:59:38.806312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.777327    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-envoy\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.806312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.777390    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-credential-socket\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.806312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.777442    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd6158f-d857-4148-b91d-f67591bd37dd-istiod-ca-cert\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.806312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.777467    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-token\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.806312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.777515    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9dk\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-kube-api-access-ds9dk\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.806312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.777580    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-socket\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.806312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.777682    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-data\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.806312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.777702    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-podinfo\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.815819 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.815454    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-certs\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917405 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917314    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-data\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917405 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917359    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-podinfo\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917406    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-certs\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917522    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-envoy\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917548    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-credential-socket\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917595    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd6158f-d857-4148-b91d-f67591bd37dd-istiod-ca-cert\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917620    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-token\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917890 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917677    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9dk\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-kube-api-access-ds9dk\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.917890 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.917736    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-socket\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.921452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.918854    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-data\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.921452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.918904    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-certs\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.921452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.918925    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-credential-socket\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.921452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.918943    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-socket\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.921452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.919395    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd6158f-d857-4148-b91d-f67591bd37dd-istiod-ca-cert\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.921452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.920251    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-envoy\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.921452 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.921446    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-podinfo\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.945323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.943467    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-token\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:38.945323 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:38.943708    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9dk\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-kube-api-access-ds9dk\") pod \"e2e-mt-ea117f29-openshift-default-7c448945-djmq5\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") " pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:39.221406 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:39.221374    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:39.568524 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:59:39.568495    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd6158f_d857_4148_b91d_f67591bd37dd.slice/crio-01792bcc3d630defca48d4258cf4ed0d0d074fb62b6760f2505f42c5539b35cb WatchSource:0}: Error finding container 01792bcc3d630defca48d4258cf4ed0d0d074fb62b6760f2505f42c5539b35cb: Status 404 returned error can't find the container with id 01792bcc3d630defca48d4258cf4ed0d0d074fb62b6760f2505f42c5539b35cb
Oct 30 12:59:39.569076 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:39.569050    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"]
Oct 30 12:59:39.570974 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:39.570945    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:39.571083 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:39.571010    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:39.571083 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:39.571040    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:40.575230 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:40.575197    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" event={"ID":"ecd6158f-d857-4148-b91d-f67591bd37dd","Type":"ContainerStarted","Data":"bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e"}
Oct 30 12:59:40.575613 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:40.575251    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" event={"ID":"ecd6158f-d857-4148-b91d-f67591bd37dd","Type":"ContainerStarted","Data":"01792bcc3d630defca48d4258cf4ed0d0d074fb62b6760f2505f42c5539b35cb"}
Oct 30 12:59:40.640418 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:40.640366    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" podStartSLOduration=2.640347539 podStartE2EDuration="2.640347539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:40.639844695 +0000 UTC m=+1638.695338520" watchObservedRunningTime="2026-10-30 12:59:40.640347539 +0000 UTC m=+1638.695841366"
Oct 30 12:59:40.939412 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:40.939384    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"]
Oct 30 12:59:40.953580 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:40.953548    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:40.978672 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:40.970237    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"]
Oct 30 12:59:41.048645 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.048616    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.048645 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.048647    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d429j\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-kube-api-access-d429j\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.048845 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.048668    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.048845 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.048819    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.048954 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.048849    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.048954 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.048865    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.048954 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.048880    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.049118 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.049024    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.049118 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.049077    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150273 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150241    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150432 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150275    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d429j\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-kube-api-access-d429j\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150432 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150337    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150558 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150444    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150558 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150474    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150558 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150498    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150558 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150525    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150567    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150772 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150634    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.150868 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.150817    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.151032 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.151012    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.151104 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.151039    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.151104 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.151076    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.151179 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.151122    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.152908 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.152887    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.153189 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.153174    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.174717 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.174695    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d429j\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-kube-api-access-d429j\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.174809 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.174744    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.221681 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.221614    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:41.227583 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.227555    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:41.277983 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.277953    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:41.584362 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.584243    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:41.585517 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.585494    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 12:59:41.600226 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:59:41.600196    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea91e557_d41c_4ffa_bd7a_ece09cc765f9.slice/crio-4a32e9871501142ce9b9a9e7c8014928e052bd79d4eebe84eecb837cf2bf906a WatchSource:0}: Error finding container 4a32e9871501142ce9b9a9e7c8014928e052bd79d4eebe84eecb837cf2bf906a: Status 404 returned error can't find the container with id 4a32e9871501142ce9b9a9e7c8014928e052bd79d4eebe84eecb837cf2bf906a
Oct 30 12:59:41.600981 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.600960    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"]
Oct 30 12:59:41.602936 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.602907    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:41.603021 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.602969    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:41.603060 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:41.602999    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:42.396852 ip-10-0-136-246 kubenswrapper[4429]: E1030 12:59:42.396812    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 12:59:42.598499 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:42.598466    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" event={"ID":"ea91e557-d41c-4ffa-bd7a-ece09cc765f9","Type":"ContainerStarted","Data":"d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a"}
Oct 30 12:59:42.598893 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:42.598522    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" event={"ID":"ea91e557-d41c-4ffa-bd7a-ece09cc765f9","Type":"ContainerStarted","Data":"4a32e9871501142ce9b9a9e7c8014928e052bd79d4eebe84eecb837cf2bf906a"}
Oct 30 12:59:42.676547 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:42.676420    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podStartSLOduration=2.676401396 podStartE2EDuration="2.676401396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:42.67542641 +0000 UTC m=+1640.730920235" watchObservedRunningTime="2026-10-30 12:59:42.676401396 +0000 UTC m=+1640.731895239"
Oct 30 12:59:43.278451 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:43.278419    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:43.279819 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:43.279793    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:59:43.279929 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:43.279846    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:59:44.279709 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:44.279668    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:59:44.280104 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:44.279727    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:59:45.279212 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:45.279172    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:59:45.279427 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:45.279229    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:59:46.278571 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:46.278538    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:59:46.278955 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:46.278587    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:59:46.569361 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:46.568442    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 12:59:47.278662 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:47.278628    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:59:47.279093 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:47.278702    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:59:47.557631 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:47.557545    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 12:59:48.278598 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:48.278563    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 12:59:48.278787 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:48.278628    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 12:59:49.282949 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:49.282918    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:49.283442 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:49.283313    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:49.284077 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:49.284057    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 12:59:52.269312 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:52.258020    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:52.350313 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:52.349363    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:52.360318 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:52.358374    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 12:59:52.451602 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:52.451552    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:52.603493 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:52.603394    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 12:59:53.272600 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.272558    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.344311 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.336862    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:59:53.371320 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.369371    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.371320 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.369747    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.371320 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.370058    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:59:53.371320 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.370115    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:59:53.453396 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.453353    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.453920 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.453899    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:59:53.639363 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.637515    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:53.639363 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.638195    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:59:53.982566 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:53.982114    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:54.306357 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:54.306235    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:54.314670 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:54.314579    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:54.375823 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:54.375782    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:54.377224 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:54.377192    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 12:59:54.468430 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:54.468381    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:59:54.642362 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:54.641811    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 12:59:55.024043 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:55.023933    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 12:59:55.039637 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:55.039522    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 12:59:55.050413 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:55.049163    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 12:59:55.051402 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:55.051374    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 12:59:55.087661 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:55.087617    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 12:59:56.960123 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:56.960086    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"]
Oct 30 12:59:56.964360 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:56.964342    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:56.985817 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:56.985789    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"]
Oct 30 12:59:56.994789 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:56.994769    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ea117f29-serving-cert\""
Oct 30 12:59:57.126540 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.126516    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-maas-api-tls\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.126685 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.126581    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4fh\" (UniqueName: \"kubernetes.io/projected/85ae7064-ff8d-4675-b3b3-669c01e15740-kube-api-access-jr4fh\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.126771 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.126749    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-metrics-certs\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.126826 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.126782    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae7064-ff8d-4675-b3b3-669c01e15740-openshift-service-ca\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.228141 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.228047    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-metrics-certs\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.228141 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.228099    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae7064-ff8d-4675-b3b3-669c01e15740-openshift-service-ca\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.228369 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.228297    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-maas-api-tls\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.228369 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.228347    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4fh\" (UniqueName: \"kubernetes.io/projected/85ae7064-ff8d-4675-b3b3-669c01e15740-kube-api-access-jr4fh\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.228932 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.228851    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae7064-ff8d-4675-b3b3-669c01e15740-openshift-service-ca\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.230880 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.230856    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-metrics-certs\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.231039 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.231017    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-maas-api-tls\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.254257 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.254232    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4fh\" (UniqueName: \"kubernetes.io/projected/85ae7064-ff8d-4675-b3b3-669c01e15740-kube-api-access-jr4fh\") pod \"maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.278456 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.278431    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:57.581986 ip-10-0-136-246 kubenswrapper[4429]: W1030 12:59:57.581957    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ae7064_ff8d_4675_b3b3_669c01e15740.slice/crio-8b656e7ef045991b728a685674afaa8966573a5cd01a85cc35330e27534ebabd WatchSource:0}: Error finding container 8b656e7ef045991b728a685674afaa8966573a5cd01a85cc35330e27534ebabd: Status 404 returned error can't find the container with id 8b656e7ef045991b728a685674afaa8966573a5cd01a85cc35330e27534ebabd
Oct 30 12:59:57.582338 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.582315    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"]
Oct 30 12:59:57.584502 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:57.583686    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:59:58.274879 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:58.274812    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr" event={"ID":"85ae7064-ff8d-4675-b3b3-669c01e15740","Type":"ContainerStarted","Data":"44fb3f7da4cccbfc2a6b92dd9aa0877b494cabd88a7cfe9dfae112667bf26b18"}
Oct 30 12:59:58.274879 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:58.274862    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr" event={"ID":"85ae7064-ff8d-4675-b3b3-669c01e15740","Type":"ContainerStarted","Data":"8b656e7ef045991b728a685674afaa8966573a5cd01a85cc35330e27534ebabd"}
Oct 30 12:59:58.275268 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:58.274891    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 12:59:58.333147 ip-10-0-136-246 kubenswrapper[4429]: I1030 12:59:58.333102    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr" podStartSLOduration=1.924985419 podStartE2EDuration="2.333087243s" totalImagesPullingTime="408.101824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:58.330042429 +0000 UTC m=+1656.385536253" watchObservedRunningTime="2026-10-30 12:59:58.333087243 +0000 UTC m=+1656.388581427"
Oct 30 13:00:00.443498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.443462    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"]
Oct 30 13:00:00.452817 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.452771    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:00.503107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.503078    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:00:00.503643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.503614    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"]
Oct 30 13:00:00.579846 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.579816    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce632f0-9e8e-4199-8cbb-185a38d8b656-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm\" (UID: \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:00.580029 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.580009    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2bm\" (UniqueName: \"kubernetes.io/projected/2ce632f0-9e8e-4199-8cbb-185a38d8b656-kube-api-access-4x2bm\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm\" (UID: \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:00.681774 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.681739    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce632f0-9e8e-4199-8cbb-185a38d8b656-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm\" (UID: \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:00.682225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.682194    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2bm\" (UniqueName: \"kubernetes.io/projected/2ce632f0-9e8e-4199-8cbb-185a38d8b656-kube-api-access-4x2bm\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm\" (UID: \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:00.682467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.682448    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce632f0-9e8e-4199-8cbb-185a38d8b656-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm\" (UID: \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:00.707319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.707239    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2bm\" (UniqueName: \"kubernetes.io/projected/2ce632f0-9e8e-4199-8cbb-185a38d8b656-kube-api-access-4x2bm\") pod \"maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm\" (UID: \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:00.776213 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:00.776188    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:01.118348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:01.118320    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"]
Oct 30 13:00:01.119024 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:00:01.119002    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce632f0_9e8e_4199_8cbb_185a38d8b656.slice/crio-fdffd851d378957fbc15f7ddc54f42d2ff771f966aef6495e1990a6307f1c150 WatchSource:0}: Error finding container fdffd851d378957fbc15f7ddc54f42d2ff771f966aef6495e1990a6307f1c150: Status 404 returned error can't find the container with id fdffd851d378957fbc15f7ddc54f42d2ff771f966aef6495e1990a6307f1c150
Oct 30 13:00:01.302593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:01.302557    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" event={"ID":"2ce632f0-9e8e-4199-8cbb-185a38d8b656","Type":"ContainerStarted","Data":"fdffd851d378957fbc15f7ddc54f42d2ff771f966aef6495e1990a6307f1c150"}
Oct 30 13:00:04.286930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:04.286903    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 13:00:05.330175 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:05.330136    4429 generic.go:356] "Generic (PLEG): container finished" podID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerID="e0b0fb2b74a91ea39e708543899fb37900f9b981a5942eb3611ff755f9ffe8e9" exitCode=60
Oct 30 13:00:05.330566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:05.330185    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" event={"ID":"2ce632f0-9e8e-4199-8cbb-185a38d8b656","Type":"ContainerDied","Data":"e0b0fb2b74a91ea39e708543899fb37900f9b981a5942eb3611ff755f9ffe8e9"}
Oct 30 13:00:05.330566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:05.330445    4429 scope.go:122] "RemoveContainer" containerID="e0b0fb2b74a91ea39e708543899fb37900f9b981a5942eb3611ff755f9ffe8e9"
Oct 30 13:00:06.120887 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.120861    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"]
Oct 30 13:00:06.125372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.125352    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:06.144709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.144688    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"]
Oct 30 13:00:06.154401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.154379    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2e0e28-dockercfg-gj4ll\""
Oct 30 13:00:06.154604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.154401    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2e0e28\""
Oct 30 13:00:06.161115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.161050    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkhv\" (UniqueName: \"kubernetes.io/projected/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-kube-api-access-grkhv\") pod \"payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5\" (UID: \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:06.161246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.161224    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5\" (UID: \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:06.201139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.201115    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"]
Oct 30 13:00:06.205142 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.205124    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:06.279926 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.279884    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"]
Oct 30 13:00:06.282342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.282317    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02245b5e-b256-498d-a4fa-5767df58e630-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b\" (UID: \"02245b5e-b256-498d-a4fa-5767df58e630\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:06.282682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.282657    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clks6\" (UniqueName: \"kubernetes.io/projected/02245b5e-b256-498d-a4fa-5767df58e630-kube-api-access-clks6\") pod \"payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b\" (UID: \"02245b5e-b256-498d-a4fa-5767df58e630\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:06.282855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.282838    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5\" (UID: \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:06.283071 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.283040    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkhv\" (UniqueName: \"kubernetes.io/projected/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-kube-api-access-grkhv\") pod \"payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5\" (UID: \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:06.284420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.284375    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5\" (UID: \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:06.321308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.317356    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkhv\" (UniqueName: \"kubernetes.io/projected/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-kube-api-access-grkhv\") pod \"payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5\" (UID: \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:06.369866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.368453    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" event={"ID":"2ce632f0-9e8e-4199-8cbb-185a38d8b656","Type":"ContainerStarted","Data":"70a7db16a7213a7c17ddb1dede9ec3914fbf367719cebabb1822ddfbd6432e5a"}
Oct 30 13:00:06.387386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.385152    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clks6\" (UniqueName: \"kubernetes.io/projected/02245b5e-b256-498d-a4fa-5767df58e630-kube-api-access-clks6\") pod \"payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b\" (UID: \"02245b5e-b256-498d-a4fa-5767df58e630\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:06.387386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.385440    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02245b5e-b256-498d-a4fa-5767df58e630-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b\" (UID: \"02245b5e-b256-498d-a4fa-5767df58e630\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:06.387386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.386420    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02245b5e-b256-498d-a4fa-5767df58e630-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b\" (UID: \"02245b5e-b256-498d-a4fa-5767df58e630\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:06.411343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.411311    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clks6\" (UniqueName: \"kubernetes.io/projected/02245b5e-b256-498d-a4fa-5767df58e630-kube-api-access-clks6\") pod \"payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b\" (UID: \"02245b5e-b256-498d-a4fa-5767df58e630\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:06.420230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.420182    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" podStartSLOduration=3.633564154 podStartE2EDuration="6.420162965s" totalImagesPullingTime="2.786598811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:06.413574557 +0000 UTC m=+1664.469068382" watchObservedRunningTime="2026-10-30 13:00:06.420162965 +0000 UTC m=+1664.475656835"
Oct 30 13:00:06.439365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.439331    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:06.532044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.532015    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:06.752616 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:00:06.752591    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8dc05c_4589_4ec1_a18e_9259d19a3c8b.slice/crio-02fe3926051fe8a95c03ec2b4305d4c1229ee387924d8b6bc7aff07dc8d49413 WatchSource:0}: Error finding container 02fe3926051fe8a95c03ec2b4305d4c1229ee387924d8b6bc7aff07dc8d49413: Status 404 returned error can't find the container with id 02fe3926051fe8a95c03ec2b4305d4c1229ee387924d8b6bc7aff07dc8d49413
Oct 30 13:00:06.753097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.753076    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"]
Oct 30 13:00:06.890458 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:00:06.890401    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02245b5e_b256_498d_a4fa_5767df58e630.slice/crio-ba87cb12eb594495786f178e415577f689832edb75bfe34408621d8071a80573 WatchSource:0}: Error finding container ba87cb12eb594495786f178e415577f689832edb75bfe34408621d8071a80573: Status 404 returned error can't find the container with id ba87cb12eb594495786f178e415577f689832edb75bfe34408621d8071a80573
Oct 30 13:00:06.890889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:06.890866    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"]
Oct 30 13:00:07.376762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.376718    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b" event={"ID":"02245b5e-b256-498d-a4fa-5767df58e630","Type":"ContainerStarted","Data":"5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3"}
Oct 30 13:00:07.377232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.376793    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b" event={"ID":"02245b5e-b256-498d-a4fa-5767df58e630","Type":"ContainerStarted","Data":"ba87cb12eb594495786f178e415577f689832edb75bfe34408621d8071a80573"}
Oct 30 13:00:07.377232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.376936    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:07.379176 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.379148    4429 generic.go:356] "Generic (PLEG): container finished" podID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerID="70a7db16a7213a7c17ddb1dede9ec3914fbf367719cebabb1822ddfbd6432e5a" exitCode=60
Oct 30 13:00:07.379320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.379222    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" event={"ID":"2ce632f0-9e8e-4199-8cbb-185a38d8b656","Type":"ContainerDied","Data":"70a7db16a7213a7c17ddb1dede9ec3914fbf367719cebabb1822ddfbd6432e5a"}
Oct 30 13:00:07.379320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.379275    4429 scope.go:122] "RemoveContainer" containerID="e0b0fb2b74a91ea39e708543899fb37900f9b981a5942eb3611ff755f9ffe8e9"
Oct 30 13:00:07.379494 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.379479    4429 scope.go:122] "RemoveContainer" containerID="70a7db16a7213a7c17ddb1dede9ec3914fbf367719cebabb1822ddfbd6432e5a"
Oct 30 13:00:07.379746 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:00:07.379725    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm_odh-ai-gateway-infra(2ce632f0-9e8e-4199-8cbb-185a38d8b656)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656"
Oct 30 13:00:07.381477 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.381457    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5" event={"ID":"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b","Type":"ContainerStarted","Data":"677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183"}
Oct 30 13:00:07.381578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.381491    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5" event={"ID":"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b","Type":"ContainerStarted","Data":"02fe3926051fe8a95c03ec2b4305d4c1229ee387924d8b6bc7aff07dc8d49413"}
Oct 30 13:00:07.381578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.381551    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:07.423548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.423497    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b" podStartSLOduration=1.423485378 podStartE2EDuration="1.423485378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:07.420040223 +0000 UTC m=+1665.475534049" watchObservedRunningTime="2026-10-30 13:00:07.423485378 +0000 UTC m=+1665.478979225"
Oct 30 13:00:07.466545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:07.466505    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5" podStartSLOduration=1.466494032 podStartE2EDuration="1.466494032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:07.46458043 +0000 UTC m=+1665.520074273" watchObservedRunningTime="2026-10-30 13:00:07.466494032 +0000 UTC m=+1665.521987856"
Oct 30 13:00:08.390570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:08.390540    4429 scope.go:122] "RemoveContainer" containerID="70a7db16a7213a7c17ddb1dede9ec3914fbf367719cebabb1822ddfbd6432e5a"
Oct 30 13:00:08.391043 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:00:08.390928    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm_odh-ai-gateway-infra(2ce632f0-9e8e-4199-8cbb-185a38d8b656)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656"
Oct 30 13:00:10.396188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.396150    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"]
Oct 30 13:00:10.401534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.401515    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:10.417710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.417683    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f399fe\""
Oct 30 13:00:10.431758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.431727    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f399fe-dockercfg-qclp6\""
Oct 30 13:00:10.437261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.437230    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"]
Oct 30 13:00:10.549307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.547045    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"]
Oct 30 13:00:10.556597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.556569    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:10.577221 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.576196    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnt5d\" (UniqueName: \"kubernetes.io/projected/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-kube-api-access-lnt5d\") pod \"payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz\" (UID: \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:10.577221 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.576248    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz\" (UID: \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:10.626320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.626270    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"]
Oct 30 13:00:10.678185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.678090    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8ks\" (UniqueName: \"kubernetes.io/projected/5810c994-34a4-4a38-8390-1b2e528efae0-kube-api-access-rd8ks\") pod \"payload-processing-e2e-api-b-f399fe-966c95876-z6jf4\" (UID: \"5810c994-34a4-4a38-8390-1b2e528efae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:10.678483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.678454    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5810c994-34a4-4a38-8390-1b2e528efae0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f399fe-966c95876-z6jf4\" (UID: \"5810c994-34a4-4a38-8390-1b2e528efae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:10.678597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.678526    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnt5d\" (UniqueName: \"kubernetes.io/projected/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-kube-api-access-lnt5d\") pod \"payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz\" (UID: \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:10.678597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.678550    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz\" (UID: \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:10.679375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.679352    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz\" (UID: \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:10.704076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.704037    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnt5d\" (UniqueName: \"kubernetes.io/projected/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-kube-api-access-lnt5d\") pod \"payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz\" (UID: \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:10.718161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.718128    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:10.780038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.780007    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5810c994-34a4-4a38-8390-1b2e528efae0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f399fe-966c95876-z6jf4\" (UID: \"5810c994-34a4-4a38-8390-1b2e528efae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:10.780173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.780098    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8ks\" (UniqueName: \"kubernetes.io/projected/5810c994-34a4-4a38-8390-1b2e528efae0-kube-api-access-rd8ks\") pod \"payload-processing-e2e-api-b-f399fe-966c95876-z6jf4\" (UID: \"5810c994-34a4-4a38-8390-1b2e528efae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:10.780860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.780839    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5810c994-34a4-4a38-8390-1b2e528efae0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f399fe-966c95876-z6jf4\" (UID: \"5810c994-34a4-4a38-8390-1b2e528efae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:10.813158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.813124    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8ks\" (UniqueName: \"kubernetes.io/projected/5810c994-34a4-4a38-8390-1b2e528efae0-kube-api-access-rd8ks\") pod \"payload-processing-e2e-api-b-f399fe-966c95876-z6jf4\" (UID: \"5810c994-34a4-4a38-8390-1b2e528efae0\") " pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:10.904089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:10.904060    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:11.028398 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:00:11.028360    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4b65bb_cb5a_4998_9532_735fcfd0a6c4.slice/crio-1c58f32bf53f059ed10e8f24b9b34456e4c037c983a4c4acdff45b76382c4d67 WatchSource:0}: Error finding container 1c58f32bf53f059ed10e8f24b9b34456e4c037c983a4c4acdff45b76382c4d67: Status 404 returned error can't find the container with id 1c58f32bf53f059ed10e8f24b9b34456e4c037c983a4c4acdff45b76382c4d67
Oct 30 13:00:11.030840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.030811    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"]
Oct 30 13:00:11.220540 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:00:11.220509    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5810c994_34a4_4a38_8390_1b2e528efae0.slice/crio-3afef3c751709d85d6428d7d132f07b4c4524f1a7386bd3e34c6a4b9a08de9f3 WatchSource:0}: Error finding container 3afef3c751709d85d6428d7d132f07b4c4524f1a7386bd3e34c6a4b9a08de9f3: Status 404 returned error can't find the container with id 3afef3c751709d85d6428d7d132f07b4c4524f1a7386bd3e34c6a4b9a08de9f3
Oct 30 13:00:11.221042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.220983    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"]
Oct 30 13:00:11.417855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.417818    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz" event={"ID":"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4","Type":"ContainerStarted","Data":"1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33"}
Oct 30 13:00:11.418342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.417889    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:11.418342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.417914    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz" event={"ID":"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4","Type":"ContainerStarted","Data":"1c58f32bf53f059ed10e8f24b9b34456e4c037c983a4c4acdff45b76382c4d67"}
Oct 30 13:00:11.419499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.419472    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4" event={"ID":"5810c994-34a4-4a38-8390-1b2e528efae0","Type":"ContainerStarted","Data":"58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0"}
Oct 30 13:00:11.419619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.419516    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4" event={"ID":"5810c994-34a4-4a38-8390-1b2e528efae0","Type":"ContainerStarted","Data":"3afef3c751709d85d6428d7d132f07b4c4524f1a7386bd3e34c6a4b9a08de9f3"}
Oct 30 13:00:11.419619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.419541    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:11.463379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.463313    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz" podStartSLOduration=1.463293615 podStartE2EDuration="1.463293615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:11.461411673 +0000 UTC m=+1669.516905497" watchObservedRunningTime="2026-10-30 13:00:11.463293615 +0000 UTC m=+1669.518787440"
Oct 30 13:00:11.507803 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:11.507679    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4" podStartSLOduration=1.5076593150000002 podStartE2EDuration="1.507659315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:11.503925208 +0000 UTC m=+1669.559419032" watchObservedRunningTime="2026-10-30 13:00:11.507659315 +0000 UTC m=+1669.563153138"
Oct 30 13:00:12.371884 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:00:12.371852    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:00:18.076315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.073769    4429 scope.go:122] "RemoveContainer" containerID="70a7db16a7213a7c17ddb1dede9ec3914fbf367719cebabb1822ddfbd6432e5a"
Oct 30 13:00:18.329534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.329467    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"]
Oct 30 13:00:18.335217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.335194    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.389139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.364275    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2e0e28-serving-cert\""
Oct 30 13:00:18.389139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.388578    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrllg\" (UniqueName: \"kubernetes.io/projected/c6df557b-b44f-4ba0-861f-f677c73103be-kube-api-access-qrllg\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.389139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.388789    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-metrics-certs\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.389139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.388946    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6df557b-b44f-4ba0-861f-f677c73103be-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.389139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.389001    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.396486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.395968    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:00:18.398013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.397400    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:00:18.400860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.400345    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"]
Oct 30 13:00:18.490692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.490655    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-metrics-certs\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.490794 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.490782    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6df557b-b44f-4ba0-861f-f677c73103be-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.490859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.490843    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.490941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.490929    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrllg\" (UniqueName: \"kubernetes.io/projected/c6df557b-b44f-4ba0-861f-f677c73103be-kube-api-access-qrllg\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.491149 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:00:18.491131    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-serving-cert: secret "maas-api-e2e-api-a-2e0e28-serving-cert" not found
Oct 30 13:00:18.491230 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:00:18.491207    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls podName:c6df557b-b44f-4ba0-861f-f677c73103be nodeName:}" failed. No retries permitted until 2026-10-30 13:00:18.991186541 +0000 UTC m=+1677.046680349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls") pod "maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz" (UID: "c6df557b-b44f-4ba0-861f-f677c73103be") : secret "maas-api-e2e-api-a-2e0e28-serving-cert" not found
Oct 30 13:00:18.491899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.491877    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6df557b-b44f-4ba0-861f-f677c73103be-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.493492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.493472    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-metrics-certs\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.518870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.518842    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrllg\" (UniqueName: \"kubernetes.io/projected/c6df557b-b44f-4ba0-861f-f677c73103be-kube-api-access-qrllg\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:18.998938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:18.998904    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:19.001440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:19.001419    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls\") pod \"maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:19.266306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:19.263001    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:19.506803 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:19.506708    4429 generic.go:356] "Generic (PLEG): container finished" podID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerID="9f79b3660994291d8e55a5b13bee2265a236e0feb3860e403259010d9e56e3e3" exitCode=60
Oct 30 13:00:19.506965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:19.506794    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" event={"ID":"2ce632f0-9e8e-4199-8cbb-185a38d8b656","Type":"ContainerDied","Data":"9f79b3660994291d8e55a5b13bee2265a236e0feb3860e403259010d9e56e3e3"}
Oct 30 13:00:19.506965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:19.506862    4429 scope.go:122] "RemoveContainer" containerID="70a7db16a7213a7c17ddb1dede9ec3914fbf367719cebabb1822ddfbd6432e5a"
Oct 30 13:00:19.507400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:19.507378    4429 scope.go:122] "RemoveContainer" containerID="9f79b3660994291d8e55a5b13bee2265a236e0feb3860e403259010d9e56e3e3"
Oct 30 13:00:19.507823 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:00:19.507791    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm_odh-ai-gateway-infra(2ce632f0-9e8e-4199-8cbb-185a38d8b656)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656"
Oct 30 13:00:19.683636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:19.683590    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"]
Oct 30 13:00:19.690436 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:00:19.690407    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6df557b_b44f_4ba0_861f_f677c73103be.slice/crio-ba5b48bf23e3191ba607d6f5e67315082104db66cbf2ad946fca32c275ad6244 WatchSource:0}: Error finding container ba5b48bf23e3191ba607d6f5e67315082104db66cbf2ad946fca32c275ad6244: Status 404 returned error can't find the container with id ba5b48bf23e3191ba607d6f5e67315082104db66cbf2ad946fca32c275ad6244
Oct 30 13:00:20.514836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.514798    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz" event={"ID":"c6df557b-b44f-4ba0-861f-f677c73103be","Type":"ContainerStarted","Data":"1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a"}
Oct 30 13:00:20.515348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.514979    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:20.515348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.515015    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz" event={"ID":"c6df557b-b44f-4ba0-861f-f677c73103be","Type":"ContainerStarted","Data":"ba5b48bf23e3191ba607d6f5e67315082104db66cbf2ad946fca32c275ad6244"}
Oct 30 13:00:20.579872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.579792    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"]
Oct 30 13:00:20.583350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.583304    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz" podStartSLOduration=2.203259008 podStartE2EDuration="2.58327618s" totalImagesPullingTime="380.017172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:20.575228145 +0000 UTC m=+1678.630721971" watchObservedRunningTime="2026-10-30 13:00:20.58327618 +0000 UTC m=+1678.638770004"
Oct 30 13:00:20.823958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.823808    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:20.932233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.932206    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce632f0-9e8e-4199-8cbb-185a38d8b656-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce632f0-9e8e-4199-8cbb-185a38d8b656-maas-api-service-ca\") pod \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\" (UID: \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\") "
Oct 30 13:00:20.932444 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.932355    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce632f0-9e8e-4199-8cbb-185a38d8b656-kube-api-access-4x2bm\" (UniqueName: \"kubernetes.io/projected/2ce632f0-9e8e-4199-8cbb-185a38d8b656-kube-api-access-4x2bm\") pod \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\" (UID: \"2ce632f0-9e8e-4199-8cbb-185a38d8b656\") "
Oct 30 13:00:20.932647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.932619    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce632f0-9e8e-4199-8cbb-185a38d8b656-maas-api-service-ca" pod "2ce632f0-9e8e-4199-8cbb-185a38d8b656" (UID: "2ce632f0-9e8e-4199-8cbb-185a38d8b656"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:00:20.933019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.933000    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ce632f0-9e8e-4199-8cbb-185a38d8b656-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:00:20.934589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:20.934571    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce632f0-9e8e-4199-8cbb-185a38d8b656-kube-api-access-4x2bm" pod "2ce632f0-9e8e-4199-8cbb-185a38d8b656" (UID: "2ce632f0-9e8e-4199-8cbb-185a38d8b656"). InnerVolumeSpecName "kube-api-access-4x2bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:00:21.034983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:21.034910    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x2bm\" (UniqueName: \"kubernetes.io/projected/2ce632f0-9e8e-4199-8cbb-185a38d8b656-kube-api-access-4x2bm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:00:21.524112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:21.524079    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"
Oct 30 13:00:21.524615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:21.524113    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm" event={"ID":"2ce632f0-9e8e-4199-8cbb-185a38d8b656","Type":"ContainerDied","Data":"fdffd851d378957fbc15f7ddc54f42d2ff771f966aef6495e1990a6307f1c150"}
Oct 30 13:00:21.524615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:21.524214    4429 scope.go:122] "RemoveContainer" containerID="9f79b3660994291d8e55a5b13bee2265a236e0feb3860e403259010d9e56e3e3"
Oct 30 13:00:21.604191 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:21.604163    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"]
Oct 30 13:00:21.617855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:21.617833    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e652ca-29889420-bk6cm"]
Oct 30 13:00:22.078429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:22.078396    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" path="/var/lib/kubelet/pods/2ce632f0-9e8e-4199-8cbb-185a38d8b656/volumes"
Oct 30 13:00:22.430206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:22.430119    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:00:22.430206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:22.430203    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:00:23.666793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.666758    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"]
Oct 30 13:00:23.667839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.667818    4429 cpu_manager.go:414] "containerMap: removing container" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:00:23.667895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.667842    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:00:23.667895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.667866    4429 cpu_manager.go:414] "containerMap: removing container" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:00:23.667895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.667876    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:00:23.668000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.667895    4429 cpu_manager.go:414] "containerMap: removing container" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:00:23.668000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.667903    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:00:23.668145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.668131    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:00:23.668176 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.668151    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:00:23.674260 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.674233    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.696683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.696222    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"]
Oct 30 13:00:23.700516 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.700492    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f399fe-serving-cert\""
Oct 30 13:00:23.810937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.810902    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz79h\" (UniqueName: \"kubernetes.io/projected/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-kube-api-access-wz79h\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.811108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.811077    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-maas-api-tls\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.811253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.811127    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-metrics-certs\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.811324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.811252    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-openshift-service-ca\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.913247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.913213    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz79h\" (UniqueName: \"kubernetes.io/projected/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-kube-api-access-wz79h\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.913490 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.913441    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-maas-api-tls\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.913590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.913500    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-metrics-certs\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.913590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.913544    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-openshift-service-ca\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.914640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.914612    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-openshift-service-ca\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.917104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.917029    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-maas-api-tls\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.920854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.920830    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-metrics-certs\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.945004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.944974    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz79h\" (UniqueName: \"kubernetes.io/projected/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-kube-api-access-wz79h\") pod \"maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:23.992163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:23.992131    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:24.303819 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:00:24.303788    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd1ab22_a53d_4e0b_b230_e4a49850aa53.slice/crio-a9e754786dfd3f2362d677ca34f62fd0cdf6058452c89a5b0574e76b600b31da WatchSource:0}: Error finding container a9e754786dfd3f2362d677ca34f62fd0cdf6058452c89a5b0574e76b600b31da: Status 404 returned error can't find the container with id a9e754786dfd3f2362d677ca34f62fd0cdf6058452c89a5b0574e76b600b31da
Oct 30 13:00:24.304823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:24.304800    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"]
Oct 30 13:00:24.550626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:24.550549    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9" event={"ID":"0fd1ab22-a53d-4e0b-b230-e4a49850aa53","Type":"ContainerStarted","Data":"a9e754786dfd3f2362d677ca34f62fd0cdf6058452c89a5b0574e76b600b31da"}
Oct 30 13:00:25.563446 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:25.563405    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9" event={"ID":"0fd1ab22-a53d-4e0b-b230-e4a49850aa53","Type":"ContainerStarted","Data":"14f39229e579efbd4a3eafe99be60d04319c309fba0c7f3100a38cea5763bd2b"}
Oct 30 13:00:25.564492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:25.563547    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:25.611673 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:25.611619    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9" podStartSLOduration=2.2971296150000002 podStartE2EDuration="2.611606395s" totalImagesPullingTime="314.47678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:25.609487116 +0000 UTC m=+1683.664980945" watchObservedRunningTime="2026-10-30 13:00:25.611606395 +0000 UTC m=+1683.667100219"
Oct 30 13:00:26.530383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:26.530356    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:00:26.571799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:26.571768    4429 generic.go:356] "Generic (PLEG): container finished" podID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerID="bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e" exitCode=0
Oct 30 13:00:26.572243 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:26.571844    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" event={"ID":"ecd6158f-d857-4148-b91d-f67591bd37dd","Type":"ContainerDied","Data":"bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e"}
Oct 30 13:00:26.572243 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:26.572178    4429 scope.go:122] "RemoveContainer" containerID="bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e"
Oct 30 13:00:26.573473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:26.573448    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:26.573585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:26.573524    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:26.573585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:26.573571    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:26.584135 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:26.584120    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 13:00:27.581487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:27.581453    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" event={"ID":"ecd6158f-d857-4148-b91d-f67591bd37dd","Type":"ContainerStarted","Data":"5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918"}
Oct 30 13:00:28.222062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:28.222034    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 13:00:28.223360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:28.223336    4429 patch_prober.go:29] interesting pod/e2e-mt-ea117f29-openshift-default-7c448945-djmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:00:28.223488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:28.223380    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:00:29.222313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:29.222258    4429 patch_prober.go:29] interesting pod/e2e-mt-ea117f29-openshift-default-7c448945-djmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:00:29.222706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:29.222317    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:00:30.222459 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:30.222423    4429 patch_prober.go:29] interesting pod/e2e-mt-ea117f29-openshift-default-7c448945-djmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:00:30.222933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:30.222494    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:00:31.221786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:31.221747    4429 patch_prober.go:29] interesting pod/e2e-mt-ea117f29-openshift-default-7c448945-djmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:00:31.221986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:31.221821    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:00:31.577968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:31.577898    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:00:32.222563 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:32.222531    4429 patch_prober.go:29] interesting pod/e2e-mt-ea117f29-openshift-default-7c448945-djmq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:00:32.222756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:32.222579    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:00:33.225853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:33.225821    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 13:00:33.226243 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:33.226222    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 13:00:33.226810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:33.226795    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 13:00:42.372896 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:00:42.372861    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:00:57.361750 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:57.361721    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"]
Oct 30 13:00:57.395451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:00:57.395418    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"]
Oct 30 13:01:01.033456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.033421    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"]
Oct 30 13:01:01.034197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.034182    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce632f0-9e8e-4199-8cbb-185a38d8b656" containerName="cleanup"
Oct 30 13:01:01.037831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.037813    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.061451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.061428    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4561d2-serving-cert\""
Oct 30 13:01:01.067218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.067188    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"]
Oct 30 13:01:01.179240 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.179205    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd746c0f-e600-46b8-b983-26262de5c80a-openshift-service-ca\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.179410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.179337    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-maas-api-tls\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.179473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.179456    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62d8p\" (UniqueName: \"kubernetes.io/projected/fd746c0f-e600-46b8-b983-26262de5c80a-kube-api-access-62d8p\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.179507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.179480    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-metrics-certs\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.280985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.280955    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62d8p\" (UniqueName: \"kubernetes.io/projected/fd746c0f-e600-46b8-b983-26262de5c80a-kube-api-access-62d8p\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.281152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.280989    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-metrics-certs\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.281152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.281090    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd746c0f-e600-46b8-b983-26262de5c80a-openshift-service-ca\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.281152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.281146    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-maas-api-tls\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.281930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.281847    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd746c0f-e600-46b8-b983-26262de5c80a-openshift-service-ca\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.283765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.283716    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-maas-api-tls\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.283862 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.283796    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-metrics-certs\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.306299 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.306264    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62d8p\" (UniqueName: \"kubernetes.io/projected/fd746c0f-e600-46b8-b983-26262de5c80a-kube-api-access-62d8p\") pod \"maas-api-e2e-api-b-4561d2-559fff9f95-z426g\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.354044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.354014    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:01.681421 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:01:01.681383    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd746c0f_e600_46b8_b983_26262de5c80a.slice/crio-8292b3346b1db5792a4da6f2cd7726930697fee8bfe5f7e9224932f39ef99bf7 WatchSource:0}: Error finding container 8292b3346b1db5792a4da6f2cd7726930697fee8bfe5f7e9224932f39ef99bf7: Status 404 returned error can't find the container with id 8292b3346b1db5792a4da6f2cd7726930697fee8bfe5f7e9224932f39ef99bf7
Oct 30 13:01:01.682029 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.682006    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"]
Oct 30 13:01:01.830669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:01.830641    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g" event={"ID":"fd746c0f-e600-46b8-b983-26262de5c80a","Type":"ContainerStarted","Data":"8292b3346b1db5792a4da6f2cd7726930697fee8bfe5f7e9224932f39ef99bf7"}
Oct 30 13:01:02.362163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.362076    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst" podUID="33e511ae-0712-46f3-9088-2c763a5f2c11" containerName="payload-processing" containerID="cri-o://a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b" gracePeriod=25
Oct 30 13:01:02.396446 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.396414    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr" podUID="976c3d26-4b6f-4b98-9901-81c197bc7815" containerName="payload-pre-processing" containerID="cri-o://8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc" gracePeriod=25
Oct 30 13:01:02.697440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.697414    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 13:01:02.715913 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.715892    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 13:01:02.810437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.810409    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e511ae-0712-46f3-9088-2c763a5f2c11-kube-api-access-6pprw\" (UniqueName: \"kubernetes.io/projected/33e511ae-0712-46f3-9088-2c763a5f2c11-kube-api-access-6pprw\") pod \"33e511ae-0712-46f3-9088-2c763a5f2c11\" (UID: \"33e511ae-0712-46f3-9088-2c763a5f2c11\") "
Oct 30 13:01:02.810727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.810575    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e511ae-0712-46f3-9088-2c763a5f2c11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e511ae-0712-46f3-9088-2c763a5f2c11-plugins-config-volume\") pod \"33e511ae-0712-46f3-9088-2c763a5f2c11\" (UID: \"33e511ae-0712-46f3-9088-2c763a5f2c11\") "
Oct 30 13:01:02.810727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.810681    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976c3d26-4b6f-4b98-9901-81c197bc7815-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c3d26-4b6f-4b98-9901-81c197bc7815-plugins-config-volume\") pod \"976c3d26-4b6f-4b98-9901-81c197bc7815\" (UID: \"976c3d26-4b6f-4b98-9901-81c197bc7815\") "
Oct 30 13:01:02.810871 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.810821    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976c3d26-4b6f-4b98-9901-81c197bc7815-kube-api-access-vpxct\" (UniqueName: \"kubernetes.io/projected/976c3d26-4b6f-4b98-9901-81c197bc7815-kube-api-access-vpxct\") pod \"976c3d26-4b6f-4b98-9901-81c197bc7815\" (UID: \"976c3d26-4b6f-4b98-9901-81c197bc7815\") "
Oct 30 13:01:02.811102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.811077    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e511ae-0712-46f3-9088-2c763a5f2c11-plugins-config-volume" pod "33e511ae-0712-46f3-9088-2c763a5f2c11" (UID: "33e511ae-0712-46f3-9088-2c763a5f2c11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:02.811614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.811590    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e511ae-0712-46f3-9088-2c763a5f2c11-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:02.811717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.811608    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976c3d26-4b6f-4b98-9901-81c197bc7815-plugins-config-volume" pod "976c3d26-4b6f-4b98-9901-81c197bc7815" (UID: "976c3d26-4b6f-4b98-9901-81c197bc7815"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:02.813710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.813683    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e511ae-0712-46f3-9088-2c763a5f2c11-kube-api-access-6pprw" pod "33e511ae-0712-46f3-9088-2c763a5f2c11" (UID: "33e511ae-0712-46f3-9088-2c763a5f2c11"). InnerVolumeSpecName "kube-api-access-6pprw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:02.814232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.814208    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976c3d26-4b6f-4b98-9901-81c197bc7815-kube-api-access-vpxct" pod "976c3d26-4b6f-4b98-9901-81c197bc7815" (UID: "976c3d26-4b6f-4b98-9901-81c197bc7815"). InnerVolumeSpecName "kube-api-access-vpxct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:02.867423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.867386    4429 generic.go:356] "Generic (PLEG): container finished" podID="33e511ae-0712-46f3-9088-2c763a5f2c11" containerID="a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b" exitCode=0
Oct 30 13:01:02.867627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.867424    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst" event={"ID":"33e511ae-0712-46f3-9088-2c763a5f2c11","Type":"ContainerDied","Data":"a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b"}
Oct 30 13:01:02.867627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.867492    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst" event={"ID":"33e511ae-0712-46f3-9088-2c763a5f2c11","Type":"ContainerDied","Data":"dd0280a3262bbb952cfed9bcce9b4ed8b8652113554cb51ff011560c9451c65f"}
Oct 30 13:01:02.867627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.867442    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"
Oct 30 13:01:02.867627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.867521    4429 scope.go:122] "RemoveContainer" containerID="a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b"
Oct 30 13:01:02.869833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.869807    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g" event={"ID":"fd746c0f-e600-46b8-b983-26262de5c80a","Type":"ContainerStarted","Data":"493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2"}
Oct 30 13:01:02.869949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.869872    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:02.871211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.871185    4429 generic.go:356] "Generic (PLEG): container finished" podID="976c3d26-4b6f-4b98-9901-81c197bc7815" containerID="8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc" exitCode=0
Oct 30 13:01:02.871373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.871220    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"
Oct 30 13:01:02.871373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.871231    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr" event={"ID":"976c3d26-4b6f-4b98-9901-81c197bc7815","Type":"ContainerDied","Data":"8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc"}
Oct 30 13:01:02.871373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.871321    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr" event={"ID":"976c3d26-4b6f-4b98-9901-81c197bc7815","Type":"ContainerDied","Data":"79d07451d1cb22ba5ad60d62f2a4a7ce10f72a93260258351cf6b458e493e029"}
Oct 30 13:01:02.881070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.881047    4429 scope.go:122] "RemoveContainer" containerID="a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b"
Oct 30 13:01:02.881434 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:02.881406    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b\": container with ID starting with a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b not found: ID does not exist" containerID="a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b"
Oct 30 13:01:02.881548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.881440    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b"} err="failed to get container status \"a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b\": rpc error: code = NotFound desc = could not find container \"a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b\": container with ID starting with a41d4bb9db34b098fa4c771149ff9ab6d3b95eec6b2079804c1558141c4a0e3b not found: ID does not exist"
Oct 30 13:01:02.881548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.881461    4429 scope.go:122] "RemoveContainer" containerID="8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc"
Oct 30 13:01:02.892597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.892571    4429 scope.go:122] "RemoveContainer" containerID="8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc"
Oct 30 13:01:02.893071 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:02.893045    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc\": container with ID starting with 8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc not found: ID does not exist" containerID="8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc"
Oct 30 13:01:02.893166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.893080    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc"} err="failed to get container status \"8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc\": rpc error: code = NotFound desc = could not find container \"8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc\": container with ID starting with 8790a62c05ea484517daefd16568585b0ae5ecd6912cc862762ee6ccf3e0bcfc not found: ID does not exist"
Oct 30 13:01:02.912894 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.912873    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxct\" (UniqueName: \"kubernetes.io/projected/976c3d26-4b6f-4b98-9901-81c197bc7815-kube-api-access-vpxct\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:02.912894 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.912894    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pprw\" (UniqueName: \"kubernetes.io/projected/33e511ae-0712-46f3-9088-2c763a5f2c11-kube-api-access-6pprw\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:02.913030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.912905    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c3d26-4b6f-4b98-9901-81c197bc7815-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:02.926103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.926051    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"]
Oct 30 13:01:02.929877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.929854    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-87cae1-f4ff486c5-npnrr"]
Oct 30 13:01:02.966313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.966271    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"]
Oct 30 13:01:02.970603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:02.970585    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-87cae1-75d9855d76-mgtst"]
Oct 30 13:01:03.014664 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:03.014619    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g" podStartSLOduration=1.686746414 podStartE2EDuration="2.01460743s" totalImagesPullingTime="327.861016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:03.012059531 +0000 UTC m=+1721.067553354" watchObservedRunningTime="2026-10-30 13:01:03.01460743 +0000 UTC m=+1721.070101255"
Oct 30 13:01:04.075814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:04.075783    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e511ae-0712-46f3-9088-2c763a5f2c11" path="/var/lib/kubelet/pods/33e511ae-0712-46f3-9088-2c763a5f2c11/volumes"
Oct 30 13:01:04.076209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:04.076194    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976c3d26-4b6f-4b98-9901-81c197bc7815" path="/var/lib/kubelet/pods/976c3d26-4b6f-4b98-9901-81c197bc7815/volumes"
Oct 30 13:01:08.890228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:08.890194    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:01:12.372741 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:12.372692    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:01:32.421663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:32.421631    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"]
Oct 30 13:01:32.455108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:32.455078    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"]
Oct 30 13:01:37.422421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.422334    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b" podUID="02245b5e-b256-498d-a4fa-5767df58e630" containerName="payload-processing" containerID="cri-o://5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3" gracePeriod=25
Oct 30 13:01:37.455943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.455913    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5" podUID="dc8dc05c-4589-4ec1-a18e-9259d19a3c8b" containerName="payload-pre-processing" containerID="cri-o://677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183" gracePeriod=25
Oct 30 13:01:37.741160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.741131    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:01:37.801332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.793584    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02245b5e-b256-498d-a4fa-5767df58e630-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02245b5e-b256-498d-a4fa-5767df58e630-plugins-config-volume\") pod \"02245b5e-b256-498d-a4fa-5767df58e630\" (UID: \"02245b5e-b256-498d-a4fa-5767df58e630\") "
Oct 30 13:01:37.801332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.793831    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02245b5e-b256-498d-a4fa-5767df58e630-kube-api-access-clks6\" (UniqueName: \"kubernetes.io/projected/02245b5e-b256-498d-a4fa-5767df58e630-kube-api-access-clks6\") pod \"02245b5e-b256-498d-a4fa-5767df58e630\" (UID: \"02245b5e-b256-498d-a4fa-5767df58e630\") "
Oct 30 13:01:37.801332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.795321    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02245b5e-b256-498d-a4fa-5767df58e630-plugins-config-volume" pod "02245b5e-b256-498d-a4fa-5767df58e630" (UID: "02245b5e-b256-498d-a4fa-5767df58e630"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:37.809124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.809070    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02245b5e-b256-498d-a4fa-5767df58e630-kube-api-access-clks6" pod "02245b5e-b256-498d-a4fa-5767df58e630" (UID: "02245b5e-b256-498d-a4fa-5767df58e630"). InnerVolumeSpecName "kube-api-access-clks6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:37.813501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.812031    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:01:37.895872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.895837    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-plugins-config-volume\") pod \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\" (UID: \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\") "
Oct 30 13:01:37.896081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.895947    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-kube-api-access-grkhv\" (UniqueName: \"kubernetes.io/projected/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-kube-api-access-grkhv\") pod \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\" (UID: \"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b\") "
Oct 30 13:01:37.896216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.896195    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-plugins-config-volume" pod "dc8dc05c-4589-4ec1-a18e-9259d19a3c8b" (UID: "dc8dc05c-4589-4ec1-a18e-9259d19a3c8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:37.896505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.896489    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:37.896505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.896506    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02245b5e-b256-498d-a4fa-5767df58e630-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:37.896656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.896515    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clks6\" (UniqueName: \"kubernetes.io/projected/02245b5e-b256-498d-a4fa-5767df58e630-kube-api-access-clks6\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:37.898796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.898763    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-kube-api-access-grkhv" pod "dc8dc05c-4589-4ec1-a18e-9259d19a3c8b" (UID: "dc8dc05c-4589-4ec1-a18e-9259d19a3c8b"). InnerVolumeSpecName "kube-api-access-grkhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:37.998029 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:37.997962    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grkhv\" (UniqueName: \"kubernetes.io/projected/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b-kube-api-access-grkhv\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:38.160838 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.160808    4429 generic.go:356] "Generic (PLEG): container finished" podID="dc8dc05c-4589-4ec1-a18e-9259d19a3c8b" containerID="677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183" exitCode=0
Oct 30 13:01:38.161084 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.161068    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"
Oct 30 13:01:38.162941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.161622    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5" event={"ID":"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b","Type":"ContainerDied","Data":"677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183"}
Oct 30 13:01:38.162941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.161684    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5" event={"ID":"dc8dc05c-4589-4ec1-a18e-9259d19a3c8b","Type":"ContainerDied","Data":"02fe3926051fe8a95c03ec2b4305d4c1229ee387924d8b6bc7aff07dc8d49413"}
Oct 30 13:01:38.162941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.161713    4429 scope.go:122] "RemoveContainer" containerID="677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183"
Oct 30 13:01:38.164401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.164375    4429 generic.go:356] "Generic (PLEG): container finished" podID="02245b5e-b256-498d-a4fa-5767df58e630" containerID="5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3" exitCode=0
Oct 30 13:01:38.164550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.164533    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"
Oct 30 13:01:38.164662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.164457    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b" event={"ID":"02245b5e-b256-498d-a4fa-5767df58e630","Type":"ContainerDied","Data":"5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3"}
Oct 30 13:01:38.164740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.164704    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b" event={"ID":"02245b5e-b256-498d-a4fa-5767df58e630","Type":"ContainerDied","Data":"ba87cb12eb594495786f178e415577f689832edb75bfe34408621d8071a80573"}
Oct 30 13:01:38.176398 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.176277    4429 scope.go:122] "RemoveContainer" containerID="677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183"
Oct 30 13:01:38.177474 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:38.177439    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183\": container with ID starting with 677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183 not found: ID does not exist" containerID="677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183"
Oct 30 13:01:38.177582 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.177479    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183"} err="failed to get container status \"677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183\": rpc error: code = NotFound desc = could not find container \"677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183\": container with ID starting with 677cbfaa3d60734bd69a94ec1cb961a55cc4357ba463a819660b508a20be7183 not found: ID does not exist"
Oct 30 13:01:38.177582 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.177502    4429 scope.go:122] "RemoveContainer" containerID="5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3"
Oct 30 13:01:38.188224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.188202    4429 scope.go:122] "RemoveContainer" containerID="5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3"
Oct 30 13:01:38.188571 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:38.188543    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3\": container with ID starting with 5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3 not found: ID does not exist" containerID="5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3"
Oct 30 13:01:38.188692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.188579    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3"} err="failed to get container status \"5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3\": rpc error: code = NotFound desc = could not find container \"5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3\": container with ID starting with 5c693ace57aeb9393cad5de49483de2548505e007d30f17dc55474fe828b2bc3 not found: ID does not exist"
Oct 30 13:01:38.233974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.233952    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"]
Oct 30 13:01:38.245519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.245500    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e0e28-865bb67d4b-fvl9b"]
Oct 30 13:01:38.273506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.273456    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"]
Oct 30 13:01:38.279982 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:38.279962    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e0e28-6c88d56c79-5h4j5"]
Oct 30 13:01:40.076481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:40.076453    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02245b5e-b256-498d-a4fa-5767df58e630" path="/var/lib/kubelet/pods/02245b5e-b256-498d-a4fa-5767df58e630/volumes"
Oct 30 13:01:40.076939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:40.076831    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8dc05c-4589-4ec1-a18e-9259d19a3c8b" path="/var/lib/kubelet/pods/dc8dc05c-4589-4ec1-a18e-9259d19a3c8b/volumes"
Oct 30 13:01:42.373954 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:42.373921    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:01:45.418747 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:45.418716    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"]
Oct 30 13:01:45.453265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:45.453233    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"]
Oct 30 13:01:50.419170 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.419108    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4" podUID="5810c994-34a4-4a38-8390-1b2e528efae0" containerName="payload-processing" containerID="cri-o://58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0" gracePeriod=25
Oct 30 13:01:50.454620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.454587    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz" podUID="8b4b65bb-cb5a-4998-9532-735fcfd0a6c4" containerName="payload-pre-processing" containerID="cri-o://1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33" gracePeriod=25
Oct 30 13:01:50.767448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.767422    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:01:50.795875 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.795855    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:01:50.801659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.801637    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-plugins-config-volume\") pod \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\" (UID: \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\") "
Oct 30 13:01:50.801783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.801708    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-kube-api-access-lnt5d\" (UniqueName: \"kubernetes.io/projected/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-kube-api-access-lnt5d\") pod \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\" (UID: \"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4\") "
Oct 30 13:01:50.801783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.801776    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5810c994-34a4-4a38-8390-1b2e528efae0-kube-api-access-rd8ks\" (UniqueName: \"kubernetes.io/projected/5810c994-34a4-4a38-8390-1b2e528efae0-kube-api-access-rd8ks\") pod \"5810c994-34a4-4a38-8390-1b2e528efae0\" (UID: \"5810c994-34a4-4a38-8390-1b2e528efae0\") "
Oct 30 13:01:50.801944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.801929    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5810c994-34a4-4a38-8390-1b2e528efae0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5810c994-34a4-4a38-8390-1b2e528efae0-plugins-config-volume\") pod \"5810c994-34a4-4a38-8390-1b2e528efae0\" (UID: \"5810c994-34a4-4a38-8390-1b2e528efae0\") "
Oct 30 13:01:50.802082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.802054    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-plugins-config-volume" pod "8b4b65bb-cb5a-4998-9532-735fcfd0a6c4" (UID: "8b4b65bb-cb5a-4998-9532-735fcfd0a6c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:50.802382 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.802358    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5810c994-34a4-4a38-8390-1b2e528efae0-plugins-config-volume" pod "5810c994-34a4-4a38-8390-1b2e528efae0" (UID: "5810c994-34a4-4a38-8390-1b2e528efae0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:50.802532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.802519    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5810c994-34a4-4a38-8390-1b2e528efae0-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:50.802572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.802535    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:50.803960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.803941    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-kube-api-access-lnt5d" pod "8b4b65bb-cb5a-4998-9532-735fcfd0a6c4" (UID: "8b4b65bb-cb5a-4998-9532-735fcfd0a6c4"). InnerVolumeSpecName "kube-api-access-lnt5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:50.804006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.803968    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5810c994-34a4-4a38-8390-1b2e528efae0-kube-api-access-rd8ks" pod "5810c994-34a4-4a38-8390-1b2e528efae0" (UID: "5810c994-34a4-4a38-8390-1b2e528efae0"). InnerVolumeSpecName "kube-api-access-rd8ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:50.904368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.904343    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8ks\" (UniqueName: \"kubernetes.io/projected/5810c994-34a4-4a38-8390-1b2e528efae0-kube-api-access-rd8ks\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:50.904368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:50.904367    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnt5d\" (UniqueName: \"kubernetes.io/projected/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4-kube-api-access-lnt5d\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:51.266711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.266678    4429 generic.go:356] "Generic (PLEG): container finished" podID="8b4b65bb-cb5a-4998-9532-735fcfd0a6c4" containerID="1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33" exitCode=0
Oct 30 13:01:51.266893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.266723    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"
Oct 30 13:01:51.266893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.266762    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz" event={"ID":"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4","Type":"ContainerDied","Data":"1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33"}
Oct 30 13:01:51.266893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.266816    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz" event={"ID":"8b4b65bb-cb5a-4998-9532-735fcfd0a6c4","Type":"ContainerDied","Data":"1c58f32bf53f059ed10e8f24b9b34456e4c037c983a4c4acdff45b76382c4d67"}
Oct 30 13:01:51.266893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.266845    4429 scope.go:122] "RemoveContainer" containerID="1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33"
Oct 30 13:01:51.268268 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.268241    4429 generic.go:356] "Generic (PLEG): container finished" podID="5810c994-34a4-4a38-8390-1b2e528efae0" containerID="58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0" exitCode=0
Oct 30 13:01:51.268422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.268321    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4" event={"ID":"5810c994-34a4-4a38-8390-1b2e528efae0","Type":"ContainerDied","Data":"58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0"}
Oct 30 13:01:51.268422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.268359    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4" event={"ID":"5810c994-34a4-4a38-8390-1b2e528efae0","Type":"ContainerDied","Data":"3afef3c751709d85d6428d7d132f07b4c4524f1a7386bd3e34c6a4b9a08de9f3"}
Oct 30 13:01:51.268536 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.268424    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"
Oct 30 13:01:51.277609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.277586    4429 scope.go:122] "RemoveContainer" containerID="1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33"
Oct 30 13:01:51.278036 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:51.277894    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33\": container with ID starting with 1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33 not found: ID does not exist" containerID="1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33"
Oct 30 13:01:51.278036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.277920    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33"} err="failed to get container status \"1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33\": rpc error: code = NotFound desc = could not find container \"1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33\": container with ID starting with 1bac243e17959418e4e7ba3e43f8193fd3f3284bb94c79b906be6fad84d70c33 not found: ID does not exist"
Oct 30 13:01:51.278036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.277935    4429 scope.go:122] "RemoveContainer" containerID="58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0"
Oct 30 13:01:51.286588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.286572    4429 scope.go:122] "RemoveContainer" containerID="58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0"
Oct 30 13:01:51.286835 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:51.286817    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0\": container with ID starting with 58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0 not found: ID does not exist" containerID="58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0"
Oct 30 13:01:51.286889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.286843    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0"} err="failed to get container status \"58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0\": rpc error: code = NotFound desc = could not find container \"58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0\": container with ID starting with 58ffe8f901aa75e81a66f26a105990483ab20a564e1260c962901f47af8cf4d0 not found: ID does not exist"
Oct 30 13:01:51.325026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.325005    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"]
Oct 30 13:01:51.338530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.338506    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f399fe-6c5d9f9d8c-2fgcz"]
Oct 30 13:01:51.378685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.378665    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"]
Oct 30 13:01:51.384716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:51.384698    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f399fe-966c95876-z6jf4"]
Oct 30 13:01:52.079131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:52.079093    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5810c994-34a4-4a38-8390-1b2e528efae0" path="/var/lib/kubelet/pods/5810c994-34a4-4a38-8390-1b2e528efae0/volumes"
Oct 30 13:01:52.079858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:52.079835    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4b65bb-cb5a-4998-9532-735fcfd0a6c4" path="/var/lib/kubelet/pods/8b4b65bb-cb5a-4998-9532-735fcfd0a6c4/volumes"
Oct 30 13:01:56.402872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.402845    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"]
Oct 30 13:01:56.403426 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.403057    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz" podUID="c6df557b-b44f-4ba0-861f-f677c73103be" containerName="maas-api" containerID="cri-o://1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a" gracePeriod=30
Oct 30 13:01:56.524514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.524476    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz" podUID="c6df557b-b44f-4ba0-861f-f677c73103be" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.79:8443/health\": dial tcp 10.133.0.79:8443: connect: connection refused"
Oct 30 13:01:56.769011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.768983    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:01:56.918909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.918877    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6df557b-b44f-4ba0-861f-f677c73103be-kube-api-access-qrllg\" (UniqueName: \"kubernetes.io/projected/c6df557b-b44f-4ba0-861f-f677c73103be-kube-api-access-qrllg\") pod \"c6df557b-b44f-4ba0-861f-f677c73103be\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") "
Oct 30 13:01:56.919071 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.918921    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-metrics-certs\") pod \"c6df557b-b44f-4ba0-861f-f677c73103be\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") "
Oct 30 13:01:56.919071 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.919022    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls\") pod \"c6df557b-b44f-4ba0-861f-f677c73103be\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") "
Oct 30 13:01:56.919150 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.919124    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6df557b-b44f-4ba0-861f-f677c73103be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6df557b-b44f-4ba0-861f-f677c73103be-openshift-service-ca\") pod \"c6df557b-b44f-4ba0-861f-f677c73103be\" (UID: \"c6df557b-b44f-4ba0-861f-f677c73103be\") "
Oct 30 13:01:56.919829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.919809    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6df557b-b44f-4ba0-861f-f677c73103be-openshift-service-ca" pod "c6df557b-b44f-4ba0-861f-f677c73103be" (UID: "c6df557b-b44f-4ba0-861f-f677c73103be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:01:56.925469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.925426    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-metrics-certs" pod "c6df557b-b44f-4ba0-861f-f677c73103be" (UID: "c6df557b-b44f-4ba0-861f-f677c73103be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:56.925469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.925453    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6df557b-b44f-4ba0-861f-f677c73103be-kube-api-access-qrllg" pod "c6df557b-b44f-4ba0-861f-f677c73103be" (UID: "c6df557b-b44f-4ba0-861f-f677c73103be"). InnerVolumeSpecName "kube-api-access-qrllg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:01:56.936331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:56.933512    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls" pod "c6df557b-b44f-4ba0-861f-f677c73103be" (UID: "c6df557b-b44f-4ba0-861f-f677c73103be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:01:57.025942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.025847    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrllg\" (UniqueName: \"kubernetes.io/projected/c6df557b-b44f-4ba0-861f-f677c73103be-kube-api-access-qrllg\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:57.025942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.025880    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:57.025942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.025893    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6df557b-b44f-4ba0-861f-f677c73103be-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:57.025942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.025905    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6df557b-b44f-4ba0-861f-f677c73103be-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:01:57.309147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.309066    4429 generic.go:356] "Generic (PLEG): container finished" podID="c6df557b-b44f-4ba0-861f-f677c73103be" containerID="1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a" exitCode=0
Oct 30 13:01:57.309147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.309112    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"
Oct 30 13:01:57.309406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.309154    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz" event={"ID":"c6df557b-b44f-4ba0-861f-f677c73103be","Type":"ContainerDied","Data":"1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a"}
Oct 30 13:01:57.309406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.309214    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz" event={"ID":"c6df557b-b44f-4ba0-861f-f677c73103be","Type":"ContainerDied","Data":"ba5b48bf23e3191ba607d6f5e67315082104db66cbf2ad946fca32c275ad6244"}
Oct 30 13:01:57.309406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.309245    4429 scope.go:122] "RemoveContainer" containerID="1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a"
Oct 30 13:01:57.321797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.321727    4429 scope.go:122] "RemoveContainer" containerID="1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a"
Oct 30 13:01:57.322204 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:01:57.322126    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a\": container with ID starting with 1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a not found: ID does not exist" containerID="1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a"
Oct 30 13:01:57.322204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.322163    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a"} err="failed to get container status \"1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a\": rpc error: code = NotFound desc = could not find container \"1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a\": container with ID starting with 1997ace0096a966d56c67cb149f5688991f8708560ad6e6441f080d2e237954a not found: ID does not exist"
Oct 30 13:01:57.414004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.413977    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"]
Oct 30 13:01:57.430220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:57.428642    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e0e28-6ff7f4c64-xz4nz"]
Oct 30 13:01:58.075074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:01:58.075030    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6df557b-b44f-4ba0-861f-f677c73103be" path="/var/lib/kubelet/pods/c6df557b-b44f-4ba0-861f-f677c73103be/volumes"
Oct 30 13:02:00.920778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.920750    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"]
Oct 30 13:02:00.921388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921372    4429 cpu_manager.go:414] "containerMap: removing container" podUID="02245b5e-b256-498d-a4fa-5767df58e630" containerName="payload-processing"
Oct 30 13:02:00.921432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921389    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02245b5e-b256-498d-a4fa-5767df58e630" containerName="payload-processing"
Oct 30 13:02:00.921432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921408    4429 cpu_manager.go:414] "containerMap: removing container" podUID="8b4b65bb-cb5a-4998-9532-735fcfd0a6c4" containerName="payload-pre-processing"
Oct 30 13:02:00.921432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921413    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4b65bb-cb5a-4998-9532-735fcfd0a6c4" containerName="payload-pre-processing"
Oct 30 13:02:00.921432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921429    4429 cpu_manager.go:414] "containerMap: removing container" podUID="976c3d26-4b6f-4b98-9901-81c197bc7815" containerName="payload-pre-processing"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921434    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976c3d26-4b6f-4b98-9901-81c197bc7815" containerName="payload-pre-processing"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921440    4429 cpu_manager.go:414] "containerMap: removing container" podUID="dc8dc05c-4589-4ec1-a18e-9259d19a3c8b" containerName="payload-pre-processing"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921445    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8dc05c-4589-4ec1-a18e-9259d19a3c8b" containerName="payload-pre-processing"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921463    4429 cpu_manager.go:414] "containerMap: removing container" podUID="5810c994-34a4-4a38-8390-1b2e528efae0" containerName="payload-processing"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921467    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5810c994-34a4-4a38-8390-1b2e528efae0" containerName="payload-processing"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921479    4429 cpu_manager.go:414] "containerMap: removing container" podUID="33e511ae-0712-46f3-9088-2c763a5f2c11" containerName="payload-processing"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921484    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e511ae-0712-46f3-9088-2c763a5f2c11" containerName="payload-processing"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921490    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c6df557b-b44f-4ba0-861f-f677c73103be" containerName="maas-api"
Oct 30 13:02:00.921564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921494    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6df557b-b44f-4ba0-861f-f677c73103be" containerName="maas-api"
Oct 30 13:02:00.921834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921574    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="5810c994-34a4-4a38-8390-1b2e528efae0" containerName="payload-processing"
Oct 30 13:02:00.921834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921585    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e511ae-0712-46f3-9088-2c763a5f2c11" containerName="payload-processing"
Oct 30 13:02:00.921834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921593    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6df557b-b44f-4ba0-861f-f677c73103be" containerName="maas-api"
Oct 30 13:02:00.921834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921600    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="02245b5e-b256-498d-a4fa-5767df58e630" containerName="payload-processing"
Oct 30 13:02:00.921834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921607    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4b65bb-cb5a-4998-9532-735fcfd0a6c4" containerName="payload-pre-processing"
Oct 30 13:02:00.921834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921614    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="976c3d26-4b6f-4b98-9901-81c197bc7815" containerName="payload-pre-processing"
Oct 30 13:02:00.921834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.921620    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8dc05c-4589-4ec1-a18e-9259d19a3c8b" containerName="payload-pre-processing"
Oct 30 13:02:00.926322 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.926272    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:00.944664 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.944640    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"]
Oct 30 13:02:00.954482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.954456    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f60f16\""
Oct 30 13:02:00.954622 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.954510    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f60f16-dockercfg-xtncs\""
Oct 30 13:02:00.997247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:00.997216    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"]
Oct 30 13:02:01.132476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.132445    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:01.158507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.158477    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56700fe-e3f5-41ed-90eb-03d636efe96b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr\" (UID: \"c56700fe-e3f5-41ed-90eb-03d636efe96b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:01.158868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.158851    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs856\" (UniqueName: \"kubernetes.io/projected/c56700fe-e3f5-41ed-90eb-03d636efe96b-kube-api-access-cs856\") pod \"payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr\" (UID: \"c56700fe-e3f5-41ed-90eb-03d636efe96b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:01.170336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.169068    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"]
Oct 30 13:02:01.261570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.261055    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56700fe-e3f5-41ed-90eb-03d636efe96b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr\" (UID: \"c56700fe-e3f5-41ed-90eb-03d636efe96b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:01.261570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.261225    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t\" (UID: \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:01.261570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.261260    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcw9\" (UniqueName: \"kubernetes.io/projected/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-kube-api-access-7gcw9\") pod \"payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t\" (UID: \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:01.262258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.262240    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs856\" (UniqueName: \"kubernetes.io/projected/c56700fe-e3f5-41ed-90eb-03d636efe96b-kube-api-access-cs856\") pod \"payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr\" (UID: \"c56700fe-e3f5-41ed-90eb-03d636efe96b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:01.263354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.262547    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56700fe-e3f5-41ed-90eb-03d636efe96b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr\" (UID: \"c56700fe-e3f5-41ed-90eb-03d636efe96b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:01.288597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.288568    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs856\" (UniqueName: \"kubernetes.io/projected/c56700fe-e3f5-41ed-90eb-03d636efe96b-kube-api-access-cs856\") pod \"payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr\" (UID: \"c56700fe-e3f5-41ed-90eb-03d636efe96b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:01.363589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.363555    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t\" (UID: \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:01.363771 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.363594    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcw9\" (UniqueName: \"kubernetes.io/projected/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-kube-api-access-7gcw9\") pod \"payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t\" (UID: \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:01.364440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.364417    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t\" (UID: \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:01.389438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.389412    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcw9\" (UniqueName: \"kubernetes.io/projected/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-kube-api-access-7gcw9\") pod \"payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t\" (UID: \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:01.468245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.468215    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:01.541407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.541081    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:01.781797 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:01.781765    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18f2d58_52ea_405d_88a0_c7675dfdb3ea.slice/crio-b2cf891f240c6d74f15d24f9bf8d0109feb428d261b4f68eb2ed643317c3b602 WatchSource:0}: Error finding container b2cf891f240c6d74f15d24f9bf8d0109feb428d261b4f68eb2ed643317c3b602: Status 404 returned error can't find the container with id b2cf891f240c6d74f15d24f9bf8d0109feb428d261b4f68eb2ed643317c3b602
Oct 30 13:02:01.782762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.782740    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"]
Oct 30 13:02:01.854978 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:01.854954    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56700fe_e3f5_41ed_90eb_03d636efe96b.slice/crio-62373acd0b59fed4cae4b5b132068b1924d8a92dd0e18c96ee5f0a2f7ab814fd WatchSource:0}: Error finding container 62373acd0b59fed4cae4b5b132068b1924d8a92dd0e18c96ee5f0a2f7ab814fd: Status 404 returned error can't find the container with id 62373acd0b59fed4cae4b5b132068b1924d8a92dd0e18c96ee5f0a2f7ab814fd
Oct 30 13:02:01.856075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:01.856055    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"]
Oct 30 13:02:02.345646 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:02.345613    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t" event={"ID":"c18f2d58-52ea-405d-88a0-c7675dfdb3ea","Type":"ContainerStarted","Data":"1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818"}
Oct 30 13:02:02.346079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:02.345670    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t" event={"ID":"c18f2d58-52ea-405d-88a0-c7675dfdb3ea","Type":"ContainerStarted","Data":"b2cf891f240c6d74f15d24f9bf8d0109feb428d261b4f68eb2ed643317c3b602"}
Oct 30 13:02:02.346079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:02.345708    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:02.347443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:02.347418    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr" event={"ID":"c56700fe-e3f5-41ed-90eb-03d636efe96b","Type":"ContainerStarted","Data":"f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922"}
Oct 30 13:02:02.347548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:02.347463    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:02.347548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:02.347478    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr" event={"ID":"c56700fe-e3f5-41ed-90eb-03d636efe96b","Type":"ContainerStarted","Data":"62373acd0b59fed4cae4b5b132068b1924d8a92dd0e18c96ee5f0a2f7ab814fd"}
Oct 30 13:02:02.390859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:02.390818    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t" podStartSLOduration=2.390804838 podStartE2EDuration="2.390804838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:02.390035745 +0000 UTC m=+1780.445529570" watchObservedRunningTime="2026-10-30 13:02:02.390804838 +0000 UTC m=+1780.446298669"
Oct 30 13:02:02.436540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:02.436491    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr" podStartSLOduration=2.436479082 podStartE2EDuration="2.436479082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:02.434151865 +0000 UTC m=+1780.489645689" watchObservedRunningTime="2026-10-30 13:02:02.436479082 +0000 UTC m=+1780.491972906"
Oct 30 13:02:07.058653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.058586    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"]
Oct 30 13:02:07.059206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.059175    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr" podUID="85ae7064-ff8d-4675-b3b3-669c01e15740" containerName="maas-api" containerID="cri-o://44fb3f7da4cccbfc2a6b92dd9aa0877b494cabd88a7cfe9dfae112667bf26b18" gracePeriod=30
Oct 30 13:02:07.383799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.383762    4429 generic.go:356] "Generic (PLEG): container finished" podID="85ae7064-ff8d-4675-b3b3-669c01e15740" containerID="44fb3f7da4cccbfc2a6b92dd9aa0877b494cabd88a7cfe9dfae112667bf26b18" exitCode=0
Oct 30 13:02:07.383992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.383843    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr" event={"ID":"85ae7064-ff8d-4675-b3b3-669c01e15740","Type":"ContainerDied","Data":"44fb3f7da4cccbfc2a6b92dd9aa0877b494cabd88a7cfe9dfae112667bf26b18"}
Oct 30 13:02:07.383992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.383914    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr" event={"ID":"85ae7064-ff8d-4675-b3b3-669c01e15740","Type":"ContainerDied","Data":"8b656e7ef045991b728a685674afaa8966573a5cd01a85cc35330e27534ebabd"}
Oct 30 13:02:07.383992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.383943    4429 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b656e7ef045991b728a685674afaa8966573a5cd01a85cc35330e27534ebabd"
Oct 30 13:02:07.387709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.387682    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 13:02:07.556088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.556052    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ae7064-ff8d-4675-b3b3-669c01e15740-kube-api-access-jr4fh\" (UniqueName: \"kubernetes.io/projected/85ae7064-ff8d-4675-b3b3-669c01e15740-kube-api-access-jr4fh\") pod \"85ae7064-ff8d-4675-b3b3-669c01e15740\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") "
Oct 30 13:02:07.556299 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.556111    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-metrics-certs\") pod \"85ae7064-ff8d-4675-b3b3-669c01e15740\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") "
Oct 30 13:02:07.556299 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.556222    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ae7064-ff8d-4675-b3b3-669c01e15740-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae7064-ff8d-4675-b3b3-669c01e15740-openshift-service-ca\") pod \"85ae7064-ff8d-4675-b3b3-669c01e15740\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") "
Oct 30 13:02:07.556299 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.556277    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-maas-api-tls\") pod \"85ae7064-ff8d-4675-b3b3-669c01e15740\" (UID: \"85ae7064-ff8d-4675-b3b3-669c01e15740\") "
Oct 30 13:02:07.556705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.556651    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ae7064-ff8d-4675-b3b3-669c01e15740-openshift-service-ca" pod "85ae7064-ff8d-4675-b3b3-669c01e15740" (UID: "85ae7064-ff8d-4675-b3b3-669c01e15740"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:07.557084 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.557065    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae7064-ff8d-4675-b3b3-669c01e15740-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:07.561746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.560601    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-maas-api-tls" pod "85ae7064-ff8d-4675-b3b3-669c01e15740" (UID: "85ae7064-ff8d-4675-b3b3-669c01e15740"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:07.561746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.560811    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ae7064-ff8d-4675-b3b3-669c01e15740-kube-api-access-jr4fh" pod "85ae7064-ff8d-4675-b3b3-669c01e15740" (UID: "85ae7064-ff8d-4675-b3b3-669c01e15740"). InnerVolumeSpecName "kube-api-access-jr4fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:07.561746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.560894    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-metrics-certs" pod "85ae7064-ff8d-4675-b3b3-669c01e15740" (UID: "85ae7064-ff8d-4675-b3b3-669c01e15740"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:07.660514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.660433    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr4fh\" (UniqueName: \"kubernetes.io/projected/85ae7064-ff8d-4675-b3b3-669c01e15740-kube-api-access-jr4fh\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:07.660514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.660469    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:07.660514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:07.660484    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae7064-ff8d-4675-b3b3-669c01e15740-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:08.390261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:08.390189    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"
Oct 30 13:02:08.454728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:08.454694    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"]
Oct 30 13:02:08.466616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:08.466591    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ea117f29-9f66dcfc7-6cthr"]
Oct 30 13:02:09.114032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.113996    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"]
Oct 30 13:02:09.114278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.114226    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9" podUID="0fd1ab22-a53d-4e0b-b230-e4a49850aa53" containerName="maas-api" containerID="cri-o://14f39229e579efbd4a3eafe99be60d04319c309fba0c7f3100a38cea5763bd2b" gracePeriod=30
Oct 30 13:02:09.438844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.438813    4429 generic.go:356] "Generic (PLEG): container finished" podID="0fd1ab22-a53d-4e0b-b230-e4a49850aa53" containerID="14f39229e579efbd4a3eafe99be60d04319c309fba0c7f3100a38cea5763bd2b" exitCode=0
Oct 30 13:02:09.439173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.438844    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9" event={"ID":"0fd1ab22-a53d-4e0b-b230-e4a49850aa53","Type":"ContainerDied","Data":"14f39229e579efbd4a3eafe99be60d04319c309fba0c7f3100a38cea5763bd2b"}
Oct 30 13:02:09.453509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.453481    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:02:09.531108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.531072    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-maas-api-tls\") pod \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") "
Oct 30 13:02:09.531345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.531124    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-metrics-certs\") pod \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") "
Oct 30 13:02:09.531345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.531233    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-openshift-service-ca\") pod \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") "
Oct 30 13:02:09.531500 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.531410    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-kube-api-access-wz79h\" (UniqueName: \"kubernetes.io/projected/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-kube-api-access-wz79h\") pod \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\" (UID: \"0fd1ab22-a53d-4e0b-b230-e4a49850aa53\") "
Oct 30 13:02:09.531684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.531639    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-openshift-service-ca" pod "0fd1ab22-a53d-4e0b-b230-e4a49850aa53" (UID: "0fd1ab22-a53d-4e0b-b230-e4a49850aa53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:09.532436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.532404    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.533712 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.533685    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-maas-api-tls" pod "0fd1ab22-a53d-4e0b-b230-e4a49850aa53" (UID: "0fd1ab22-a53d-4e0b-b230-e4a49850aa53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:09.534474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.534426    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-kube-api-access-wz79h" pod "0fd1ab22-a53d-4e0b-b230-e4a49850aa53" (UID: "0fd1ab22-a53d-4e0b-b230-e4a49850aa53"). InnerVolumeSpecName "kube-api-access-wz79h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:09.534761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.534736    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-metrics-certs" pod "0fd1ab22-a53d-4e0b-b230-e4a49850aa53" (UID: "0fd1ab22-a53d-4e0b-b230-e4a49850aa53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:09.634076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.634043    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz79h\" (UniqueName: \"kubernetes.io/projected/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-kube-api-access-wz79h\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.634076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.634077    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:09.634387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:09.634110    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd1ab22-a53d-4e0b-b230-e4a49850aa53-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:10.078933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:10.078900    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ae7064-ff8d-4675-b3b3-669c01e15740" path="/var/lib/kubelet/pods/85ae7064-ff8d-4675-b3b3-669c01e15740/volumes"
Oct 30 13:02:10.446451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:10.446423    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"
Oct 30 13:02:10.446873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:10.446429    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9" event={"ID":"0fd1ab22-a53d-4e0b-b230-e4a49850aa53","Type":"ContainerDied","Data":"a9e754786dfd3f2362d677ca34f62fd0cdf6058452c89a5b0574e76b600b31da"}
Oct 30 13:02:10.446873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:10.446552    4429 scope.go:122] "RemoveContainer" containerID="14f39229e579efbd4a3eafe99be60d04319c309fba0c7f3100a38cea5763bd2b"
Oct 30 13:02:10.504350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:10.504321    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"]
Oct 30 13:02:10.518095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:10.518068    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f399fe-598c494cb7-m6kg9"]
Oct 30 13:02:11.466490 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.466451    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"]
Oct 30 13:02:11.467365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.467344    4429 cpu_manager.go:414] "containerMap: removing container" podUID="85ae7064-ff8d-4675-b3b3-669c01e15740" containerName="maas-api"
Oct 30 13:02:11.467365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.467365    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ae7064-ff8d-4675-b3b3-669c01e15740" containerName="maas-api"
Oct 30 13:02:11.467555 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.467385    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0fd1ab22-a53d-4e0b-b230-e4a49850aa53" containerName="maas-api"
Oct 30 13:02:11.467555 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.467392    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd1ab22-a53d-4e0b-b230-e4a49850aa53" containerName="maas-api"
Oct 30 13:02:11.467665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.467561    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ae7064-ff8d-4675-b3b3-669c01e15740" containerName="maas-api"
Oct 30 13:02:11.467665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.467576    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd1ab22-a53d-4e0b-b230-e4a49850aa53" containerName="maas-api"
Oct 30 13:02:11.471753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.471733    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.491175 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.491148    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f60f16-serving-cert\""
Oct 30 13:02:11.495456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.495430    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"]
Oct 30 13:02:11.562668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.562632    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-maas-api-tls\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.562841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.562702    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df478c39-3038-43d3-9dc3-14be3ba3f3a9-openshift-service-ca\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.562841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.562778    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-metrics-certs\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.562841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.562837    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5292w\" (UniqueName: \"kubernetes.io/projected/df478c39-3038-43d3-9dc3-14be3ba3f3a9-kube-api-access-5292w\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.664668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.664623    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-maas-api-tls\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.664872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.664734    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df478c39-3038-43d3-9dc3-14be3ba3f3a9-openshift-service-ca\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.664872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.664780    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-metrics-certs\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.664872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.664853    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5292w\" (UniqueName: \"kubernetes.io/projected/df478c39-3038-43d3-9dc3-14be3ba3f3a9-kube-api-access-5292w\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.665492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.665460    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df478c39-3038-43d3-9dc3-14be3ba3f3a9-openshift-service-ca\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.667651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.667623    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-maas-api-tls\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.667651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.667631    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-metrics-certs\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.692770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.692734    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5292w\" (UniqueName: \"kubernetes.io/projected/df478c39-3038-43d3-9dc3-14be3ba3f3a9-kube-api-access-5292w\") pod \"maas-api-e2e-api-a-f60f16-98994945-wgz6b\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:11.789439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:11.789347    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:12.080493 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:12.080415    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd1ab22-a53d-4e0b-b230-e4a49850aa53" path="/var/lib/kubelet/pods/0fd1ab22-a53d-4e0b-b230-e4a49850aa53/volumes"
Oct 30 13:02:12.102437 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:12.102410    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf478c39_3038_43d3_9dc3_14be3ba3f3a9.slice/crio-4a52baf2c811bd09051fa01ee66a5eaa7d3bdeed5e2aaa2af44f4ae33f2b2a17 WatchSource:0}: Error finding container 4a52baf2c811bd09051fa01ee66a5eaa7d3bdeed5e2aaa2af44f4ae33f2b2a17: Status 404 returned error can't find the container with id 4a52baf2c811bd09051fa01ee66a5eaa7d3bdeed5e2aaa2af44f4ae33f2b2a17
Oct 30 13:02:12.102618 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:12.102593    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"]
Oct 30 13:02:12.372115 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:02:12.372035    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:02:12.469086 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:12.469049    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b" event={"ID":"df478c39-3038-43d3-9dc3-14be3ba3f3a9","Type":"ContainerStarted","Data":"4a52baf2c811bd09051fa01ee66a5eaa7d3bdeed5e2aaa2af44f4ae33f2b2a17"}
Oct 30 13:02:13.357179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:13.357150    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:02:13.357383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:13.357213    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:02:13.482712 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:13.482677    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b" event={"ID":"df478c39-3038-43d3-9dc3-14be3ba3f3a9","Type":"ContainerStarted","Data":"0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6"}
Oct 30 13:02:13.485272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:13.485247    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:13.537146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:13.537094    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b" podStartSLOduration=2.1327101 podStartE2EDuration="2.537077999s" totalImagesPullingTime="404.367899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:13.535815702 +0000 UTC m=+1791.591309525" watchObservedRunningTime="2026-10-30 13:02:13.537077999 +0000 UTC m=+1791.592571852"
Oct 30 13:02:20.501382 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.501355    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:02:20.647169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.647137    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"]
Oct 30 13:02:20.651224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.651198    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:20.672457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.672433    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"]
Oct 30 13:02:20.719203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.719174    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ea117f29\""
Oct 30 13:02:20.719726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.719708    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ea117f29-dockercfg-m55dp\""
Oct 30 13:02:20.825414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.825314    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5kn\" (UniqueName: \"kubernetes.io/projected/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-kube-api-access-8w5kn\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k\" (UID: \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:20.825644 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.825626    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k\" (UID: \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:20.926888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.926855    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5kn\" (UniqueName: \"kubernetes.io/projected/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-kube-api-access-8w5kn\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k\" (UID: \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:20.927081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.926904    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k\" (UID: \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:20.927756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.927732    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k\" (UID: \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:20.954005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.953970    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5kn\" (UniqueName: \"kubernetes.io/projected/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-kube-api-access-8w5kn\") pod \"payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k\" (UID: \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:20.971669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:20.971640    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:21.291365 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:21.291324    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d04efdc_d379_4b14_b9e6_7ac4ec47d0c2.slice/crio-00fc89611a59a444f43ff7e24d4fa0d9921a9594f0795024c758dbf8ae9aabb8 WatchSource:0}: Error finding container 00fc89611a59a444f43ff7e24d4fa0d9921a9594f0795024c758dbf8ae9aabb8: Status 404 returned error can't find the container with id 00fc89611a59a444f43ff7e24d4fa0d9921a9594f0795024c758dbf8ae9aabb8
Oct 30 13:02:21.293516 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:21.293488    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"]
Oct 30 13:02:21.535975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:21.535938    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k" event={"ID":"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2","Type":"ContainerStarted","Data":"db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3"}
Oct 30 13:02:21.536368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:21.536029    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k" event={"ID":"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2","Type":"ContainerStarted","Data":"00fc89611a59a444f43ff7e24d4fa0d9921a9594f0795024c758dbf8ae9aabb8"}
Oct 30 13:02:21.536368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:21.536173    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:21.604593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:21.604501    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k" podStartSLOduration=1.604488134 podStartE2EDuration="1.604488134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:21.602560176 +0000 UTC m=+1799.658054000" watchObservedRunningTime="2026-10-30 13:02:21.604488134 +0000 UTC m=+1799.659981958"
Oct 30 13:02:32.545856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:32.545821    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:02:41.582334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.579608    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"]
Oct 30 13:02:41.599565 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.587929    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:41.610075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.610028    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"]
Oct 30 13:02:41.621809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.621762    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a2768f\""
Oct 30 13:02:41.621980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.621892    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a2768f-dockercfg-lrszl\""
Oct 30 13:02:41.654139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.654072    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddffcdc2-3b27-4023-bd57-18e36871e8de-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn\" (UID: \"ddffcdc2-3b27-4023-bd57-18e36871e8de\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:41.654360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.654314    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tqpd\" (UniqueName: \"kubernetes.io/projected/ddffcdc2-3b27-4023-bd57-18e36871e8de-kube-api-access-7tqpd\") pod \"payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn\" (UID: \"ddffcdc2-3b27-4023-bd57-18e36871e8de\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:41.755188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.755150    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tqpd\" (UniqueName: \"kubernetes.io/projected/ddffcdc2-3b27-4023-bd57-18e36871e8de-kube-api-access-7tqpd\") pod \"payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn\" (UID: \"ddffcdc2-3b27-4023-bd57-18e36871e8de\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:41.755433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.755264    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddffcdc2-3b27-4023-bd57-18e36871e8de-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn\" (UID: \"ddffcdc2-3b27-4023-bd57-18e36871e8de\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:41.756066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.756022    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddffcdc2-3b27-4023-bd57-18e36871e8de-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn\" (UID: \"ddffcdc2-3b27-4023-bd57-18e36871e8de\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:41.780715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.780685    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tqpd\" (UniqueName: \"kubernetes.io/projected/ddffcdc2-3b27-4023-bd57-18e36871e8de-kube-api-access-7tqpd\") pod \"payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn\" (UID: \"ddffcdc2-3b27-4023-bd57-18e36871e8de\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:41.955900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:41.955867    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:42.259751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.259720    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"]
Oct 30 13:02:42.260564 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:42.260527    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddffcdc2_3b27_4023_bd57_18e36871e8de.slice/crio-c7937ebc25df9ba4e9b0d58433ce3b34874160cf0e99a2c4153cfb2e6363f3f9 WatchSource:0}: Error finding container c7937ebc25df9ba4e9b0d58433ce3b34874160cf0e99a2c4153cfb2e6363f3f9: Status 404 returned error can't find the container with id c7937ebc25df9ba4e9b0d58433ce3b34874160cf0e99a2c4153cfb2e6363f3f9
Oct 30 13:02:42.374495 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:02:42.374454    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:02:42.509117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.509010    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11526-85778e"]
Oct 30 13:02:42.513840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.513817    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-85778e"
Oct 30 13:02:42.532329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.530149    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11526-85778e"]
Oct 30 13:02:42.532329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.530509    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ljkrn\""
Oct 30 13:02:42.564403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.564374    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7r85\" (UniqueName: \"kubernetes.io/projected/eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9-kube-api-access-c7r85\") pod \"test-curl-11526-85778e\" (UID: \"eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9\") " pod="opendatahub/test-curl-11526-85778e"
Oct 30 13:02:42.665567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.665535    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7r85\" (UniqueName: \"kubernetes.io/projected/eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9-kube-api-access-c7r85\") pod \"test-curl-11526-85778e\" (UID: \"eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9\") " pod="opendatahub/test-curl-11526-85778e"
Oct 30 13:02:42.694772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.694747    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7r85\" (UniqueName: \"kubernetes.io/projected/eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9-kube-api-access-c7r85\") pod \"test-curl-11526-85778e\" (UID: \"eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9\") " pod="opendatahub/test-curl-11526-85778e"
Oct 30 13:02:42.711211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.711179    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn" event={"ID":"ddffcdc2-3b27-4023-bd57-18e36871e8de","Type":"ContainerStarted","Data":"5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7"}
Oct 30 13:02:42.711377 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.711236    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn" event={"ID":"ddffcdc2-3b27-4023-bd57-18e36871e8de","Type":"ContainerStarted","Data":"c7937ebc25df9ba4e9b0d58433ce3b34874160cf0e99a2c4153cfb2e6363f3f9"}
Oct 30 13:02:42.711377 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.711262    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:42.804352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.804258    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn" podStartSLOduration=1.804245162 podStartE2EDuration="1.804245162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:42.802733767 +0000 UTC m=+1820.858227592" watchObservedRunningTime="2026-10-30 13:02:42.804245162 +0000 UTC m=+1820.859738985"
Oct 30 13:02:42.826054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:42.826020    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-85778e"
Oct 30 13:02:43.129316 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:43.129255    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb5f0602_ec5f_4033_8a2d_e7a7f1c87ab9.slice/crio-01bbaa0cf22c7e3f9c49704fbff6768a613e83b539f481a8f3defda364caf953 WatchSource:0}: Error finding container 01bbaa0cf22c7e3f9c49704fbff6768a613e83b539f481a8f3defda364caf953: Status 404 returned error can't find the container with id 01bbaa0cf22c7e3f9c49704fbff6768a613e83b539f481a8f3defda364caf953
Oct 30 13:02:43.129642 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:43.129621    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11526-85778e"]
Oct 30 13:02:43.747150 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:43.747090    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11526-85778e" event={"ID":"eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9","Type":"ContainerStarted","Data":"01bbaa0cf22c7e3f9c49704fbff6768a613e83b539f481a8f3defda364caf953"}
Oct 30 13:02:45.791552 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:45.791461    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11526-85778e" event={"ID":"eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9","Type":"ContainerStarted","Data":"26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d"}
Oct 30 13:02:45.834768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:45.834723    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11526-85778e" podStartSLOduration=1.484129173 podStartE2EDuration="3.83470864s" totalImagesPullingTime="2.350579467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:45.833447163 +0000 UTC m=+1823.888940989" watchObservedRunningTime="2026-10-30 13:02:45.83470864 +0000 UTC m=+1823.890202464"
Oct 30 13:02:46.837299 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:46.837239    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11526-85778e"]
Oct 30 13:02:46.842974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:46.842951    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11526-85778e"]
Oct 30 13:02:47.806106 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:47.806070    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11526-85778e" podUID="eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9" containerName="test-curl-11526-85778e" containerID="cri-o://26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d" gracePeriod=2
Oct 30 13:02:47.821492 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:02:47.821461    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11526-85778e\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9" pod="opendatahub/test-curl-11526-85778e"
Oct 30 13:02:50.017550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.017524    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-85778e"
Oct 30 13:02:50.086893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.086826    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9-kube-api-access-c7r85\" (UniqueName: \"kubernetes.io/projected/eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9-kube-api-access-c7r85\") pod \"eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9\" (UID: \"eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9\") "
Oct 30 13:02:50.089124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.089103    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9-kube-api-access-c7r85" pod "eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9" (UID: "eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9"). InnerVolumeSpecName "kube-api-access-c7r85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:50.186700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.186668    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"]
Oct 30 13:02:50.187198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.187108    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g" podUID="fd746c0f-e600-46b8-b983-26262de5c80a" containerName="maas-api" containerID="cri-o://493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2" gracePeriod=30
Oct 30 13:02:50.190185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.189039    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7r85\" (UniqueName: \"kubernetes.io/projected/eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9-kube-api-access-c7r85\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:50.521064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.521039    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:02:50.666356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.659768    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd746c0f-e600-46b8-b983-26262de5c80a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd746c0f-e600-46b8-b983-26262de5c80a-openshift-service-ca\") pod \"fd746c0f-e600-46b8-b983-26262de5c80a\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") "
Oct 30 13:02:50.666356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.659857    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd746c0f-e600-46b8-b983-26262de5c80a-kube-api-access-62d8p\" (UniqueName: \"kubernetes.io/projected/fd746c0f-e600-46b8-b983-26262de5c80a-kube-api-access-62d8p\") pod \"fd746c0f-e600-46b8-b983-26262de5c80a\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") "
Oct 30 13:02:50.666356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.659913    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-maas-api-tls\") pod \"fd746c0f-e600-46b8-b983-26262de5c80a\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") "
Oct 30 13:02:50.666356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.659973    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-metrics-certs\") pod \"fd746c0f-e600-46b8-b983-26262de5c80a\" (UID: \"fd746c0f-e600-46b8-b983-26262de5c80a\") "
Oct 30 13:02:50.666356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.662721    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd746c0f-e600-46b8-b983-26262de5c80a-openshift-service-ca" pod "fd746c0f-e600-46b8-b983-26262de5c80a" (UID: "fd746c0f-e600-46b8-b983-26262de5c80a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:02:50.706274 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.706219    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-metrics-certs" pod "fd746c0f-e600-46b8-b983-26262de5c80a" (UID: "fd746c0f-e600-46b8-b983-26262de5c80a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:50.708267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.707405    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-maas-api-tls" pod "fd746c0f-e600-46b8-b983-26262de5c80a" (UID: "fd746c0f-e600-46b8-b983-26262de5c80a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:02:50.708267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.707516    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd746c0f-e600-46b8-b983-26262de5c80a-kube-api-access-62d8p" pod "fd746c0f-e600-46b8-b983-26262de5c80a" (UID: "fd746c0f-e600-46b8-b983-26262de5c80a"). InnerVolumeSpecName "kube-api-access-62d8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:02:50.767636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.767588    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd746c0f-e600-46b8-b983-26262de5c80a-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:50.767636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.767637    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62d8p\" (UniqueName: \"kubernetes.io/projected/fd746c0f-e600-46b8-b983-26262de5c80a-kube-api-access-62d8p\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:50.767916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.767654    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:50.767916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.767667    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd746c0f-e600-46b8-b983-26262de5c80a-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:02:50.829725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.829678    4429 generic.go:356] "Generic (PLEG): container finished" podID="eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9" containerID="26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d" exitCode=137
Oct 30 13:02:50.829856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.829722    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11526-85778e"
Oct 30 13:02:50.829856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.829762    4429 scope.go:122] "RemoveContainer" containerID="26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d"
Oct 30 13:02:50.831762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.831739    4429 generic.go:356] "Generic (PLEG): container finished" podID="fd746c0f-e600-46b8-b983-26262de5c80a" containerID="493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2" exitCode=0
Oct 30 13:02:50.831873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.831789    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"
Oct 30 13:02:50.831873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.831837    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g" event={"ID":"fd746c0f-e600-46b8-b983-26262de5c80a","Type":"ContainerDied","Data":"493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2"}
Oct 30 13:02:50.831985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.831885    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g" event={"ID":"fd746c0f-e600-46b8-b983-26262de5c80a","Type":"ContainerDied","Data":"8292b3346b1db5792a4da6f2cd7726930697fee8bfe5f7e9224932f39ef99bf7"}
Oct 30 13:02:50.847568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.847542    4429 scope.go:122] "RemoveContainer" containerID="26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d"
Oct 30 13:02:50.858754 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:02:50.858725    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d\": container with ID starting with 26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d not found: ID does not exist" containerID="26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d"
Oct 30 13:02:50.858849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.858773    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d"} err="failed to get container status \"26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d\": rpc error: code = NotFound desc = could not find container \"26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d\": container with ID starting with 26984787d86aa13ba953628b0f0c87acbef683e50228336804a48651c7d7a04d not found: ID does not exist"
Oct 30 13:02:50.858849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.858797    4429 scope.go:122] "RemoveContainer" containerID="493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2"
Oct 30 13:02:50.869867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.869840    4429 scope.go:122] "RemoveContainer" containerID="493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2"
Oct 30 13:02:50.870174 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:02:50.870155    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2\": container with ID starting with 493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2 not found: ID does not exist" containerID="493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2"
Oct 30 13:02:50.870261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.870184    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2"} err="failed to get container status \"493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2\": rpc error: code = NotFound desc = could not find container \"493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2\": container with ID starting with 493b059fa31904dd4221f5638c72e06fa755226c82b79e6919d628de283e8ee2 not found: ID does not exist"
Oct 30 13:02:50.908321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.908269    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"]
Oct 30 13:02:50.913521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:50.913489    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4561d2-559fff9f95-z426g"]
Oct 30 13:02:52.077484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.077454    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9" path="/var/lib/kubelet/pods/eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9/volumes"
Oct 30 13:02:52.077939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.077925    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd746c0f-e600-46b8-b983-26262de5c80a" path="/var/lib/kubelet/pods/fd746c0f-e600-46b8-b983-26262de5c80a/volumes"
Oct 30 13:02:52.503752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.503723    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"]
Oct 30 13:02:52.504339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.504324    4429 cpu_manager.go:414] "containerMap: removing container" podUID="fd746c0f-e600-46b8-b983-26262de5c80a" containerName="maas-api"
Oct 30 13:02:52.504408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.504340    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd746c0f-e600-46b8-b983-26262de5c80a" containerName="maas-api"
Oct 30 13:02:52.504408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.504376    4429 cpu_manager.go:414] "containerMap: removing container" podUID="eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9" containerName="test-curl-11526-85778e"
Oct 30 13:02:52.504408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.504381    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9" containerName="test-curl-11526-85778e"
Oct 30 13:02:52.504505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.504495    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb5f0602-ec5f-4033-8a2d-e7a7f1c87ab9" containerName="test-curl-11526-85778e"
Oct 30 13:02:52.504534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.504511    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd746c0f-e600-46b8-b983-26262de5c80a" containerName="maas-api"
Oct 30 13:02:52.509437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.509420    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:52.534220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.534171    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"]
Oct 30 13:02:52.544345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.538207    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9368ce\""
Oct 30 13:02:52.544345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.538453    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9368ce-dockercfg-gnh24\""
Oct 30 13:02:52.696094 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.686494    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"]
Oct 30 13:02:52.696849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.696487    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dm55\" (UniqueName: \"kubernetes.io/projected/b8812311-393f-47a6-b825-ea8c144e320f-kube-api-access-6dm55\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d\" (UID: \"b8812311-393f-47a6-b825-ea8c144e320f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:52.696849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.696575    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8812311-393f-47a6-b825-ea8c144e320f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d\" (UID: \"b8812311-393f-47a6-b825-ea8c144e320f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:52.701440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.699020    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"]
Oct 30 13:02:52.701440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.699148    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:52.798190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.798095    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dm55\" (UniqueName: \"kubernetes.io/projected/b8812311-393f-47a6-b825-ea8c144e320f-kube-api-access-6dm55\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d\" (UID: \"b8812311-393f-47a6-b825-ea8c144e320f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:52.798190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.798172    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8812311-393f-47a6-b825-ea8c144e320f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d\" (UID: \"b8812311-393f-47a6-b825-ea8c144e320f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:52.798482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.798213    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml\" (UID: \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:52.798482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.798325    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-kube-api-access-j9n9q\") pod \"payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml\" (UID: \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:52.798959 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.798941    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8812311-393f-47a6-b825-ea8c144e320f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d\" (UID: \"b8812311-393f-47a6-b825-ea8c144e320f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:52.837249 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.837223    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dm55\" (UniqueName: \"kubernetes.io/projected/b8812311-393f-47a6-b825-ea8c144e320f-kube-api-access-6dm55\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d\" (UID: \"b8812311-393f-47a6-b825-ea8c144e320f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:52.900360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.900329    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-kube-api-access-j9n9q\") pod \"payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml\" (UID: \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:52.900598 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.900582    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml\" (UID: \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:52.901206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.901189    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml\" (UID: \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:52.950402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.950381    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-kube-api-access-j9n9q\") pod \"payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml\" (UID: \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:52.994656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:52.994631    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"]
Oct 30 13:02:53.004780 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.004759    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.026823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.024750    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:53.026823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.025039    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a2768f-serving-cert\""
Oct 30 13:02:53.058587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.052995    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"]
Oct 30 13:02:53.107262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.102683    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-openshift-service-ca\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.107262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.102805    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-metrics-certs\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.107262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.102838    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-maas-api-tls\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.107262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.103082    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmnr\" (UniqueName: \"kubernetes.io/projected/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-kube-api-access-lrmnr\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.125551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.125522    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:53.207320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.204628    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-metrics-certs\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.207320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.204670    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-maas-api-tls\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.207320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.204769    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmnr\" (UniqueName: \"kubernetes.io/projected/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-kube-api-access-lrmnr\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.207320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.204968    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-openshift-service-ca\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.207320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.205721    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-openshift-service-ca\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.213352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.210928    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-metrics-certs\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.213352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.211376    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-maas-api-tls\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.236703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.236669    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmnr\" (UniqueName: \"kubernetes.io/projected/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-kube-api-access-lrmnr\") pod \"maas-api-e2e-api-b-a2768f-579b589d47-phmwt\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.321545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.321478    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:53.357013 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:53.356986    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4790f8_c8b3_4e43_a69a_0234bd9c3404.slice/crio-0a57547067bc2bee9e798d7e017d486595fcf42776a786cc05d083b74963b493 WatchSource:0}: Error finding container 0a57547067bc2bee9e798d7e017d486595fcf42776a786cc05d083b74963b493: Status 404 returned error can't find the container with id 0a57547067bc2bee9e798d7e017d486595fcf42776a786cc05d083b74963b493
Oct 30 13:02:53.358339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.358303    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"]
Oct 30 13:02:53.494628 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:53.494602    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8812311_393f_47a6_b825_ea8c144e320f.slice/crio-44317821955c62914903861bd8683bc904ea499ecb9130e2981cb2db5a4e2a08 WatchSource:0}: Error finding container 44317821955c62914903861bd8683bc904ea499ecb9130e2981cb2db5a4e2a08: Status 404 returned error can't find the container with id 44317821955c62914903861bd8683bc904ea499ecb9130e2981cb2db5a4e2a08
Oct 30 13:02:53.495835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.495813    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"]
Oct 30 13:02:53.635035 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:02:53.635006    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6ced03_b5d9_4419_b70e_4d4dffa7183b.slice/crio-ac5c00f7d8d2192b88107680330c3ae1f99fc0a365353486733645cf37bfba4c WatchSource:0}: Error finding container ac5c00f7d8d2192b88107680330c3ae1f99fc0a365353486733645cf37bfba4c: Status 404 returned error can't find the container with id ac5c00f7d8d2192b88107680330c3ae1f99fc0a365353486733645cf37bfba4c
Oct 30 13:02:53.635843 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.635818    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"]
Oct 30 13:02:53.750623 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.750596    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:02:53.855543 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.855499    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt" event={"ID":"fd6ced03-b5d9-4419-b70e-4d4dffa7183b","Type":"ContainerStarted","Data":"ac5c00f7d8d2192b88107680330c3ae1f99fc0a365353486733645cf37bfba4c"}
Oct 30 13:02:53.857520 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.857488    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d" event={"ID":"b8812311-393f-47a6-b825-ea8c144e320f","Type":"ContainerStarted","Data":"0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748"}
Oct 30 13:02:53.857677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.857539    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d" event={"ID":"b8812311-393f-47a6-b825-ea8c144e320f","Type":"ContainerStarted","Data":"44317821955c62914903861bd8683bc904ea499ecb9130e2981cb2db5a4e2a08"}
Oct 30 13:02:53.857677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.857587    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:02:53.859442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.859416    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml" event={"ID":"ed4790f8-c8b3-4e43-a69a-0234bd9c3404","Type":"ContainerStarted","Data":"d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669"}
Oct 30 13:02:53.859552 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.859459    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml" event={"ID":"ed4790f8-c8b3-4e43-a69a-0234bd9c3404","Type":"ContainerStarted","Data":"0a57547067bc2bee9e798d7e017d486595fcf42776a786cc05d083b74963b493"}
Oct 30 13:02:53.859552 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.859484    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:02:53.904739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.904662    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d" podStartSLOduration=1.90464989 podStartE2EDuration="1.90464989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:53.902523695 +0000 UTC m=+1831.958017520" watchObservedRunningTime="2026-10-30 13:02:53.90464989 +0000 UTC m=+1831.960143714"
Oct 30 13:02:53.946898 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:53.946858    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml" podStartSLOduration=1.9468478409999999 podStartE2EDuration="1.946847841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:53.945850433 +0000 UTC m=+1832.001344256" watchObservedRunningTime="2026-10-30 13:02:53.946847841 +0000 UTC m=+1832.002341696"
Oct 30 13:02:54.868976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:54.868940    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt" event={"ID":"fd6ced03-b5d9-4419-b70e-4d4dffa7183b","Type":"ContainerStarted","Data":"9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6"}
Oct 30 13:02:54.869512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:54.869162    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:02:54.917514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:54.917469    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt" podStartSLOduration=2.531433124 podStartE2EDuration="2.917452727s" totalImagesPullingTime="386.019603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:54.914147975 +0000 UTC m=+1832.969641798" watchObservedRunningTime="2026-10-30 13:02:54.917452727 +0000 UTC m=+1832.972946550"
Oct 30 13:02:57.656268 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:02:57.656237    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"]
Oct 30 13:03:00.881533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:00.881504    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:03:02.657256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.657204    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k" podUID="8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2" containerName="payload-pre-processing" containerID="cri-o://db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3" gracePeriod=25
Oct 30 13:03:02.956374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.952468    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:03:02.970517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.970483    4429 generic.go:356] "Generic (PLEG): container finished" podID="8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2" containerID="db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3" exitCode=0
Oct 30 13:03:02.970687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.970525    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k" event={"ID":"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2","Type":"ContainerDied","Data":"db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3"}
Oct 30 13:03:02.970687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.970549    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"
Oct 30 13:03:02.970687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.970602    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k" event={"ID":"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2","Type":"ContainerDied","Data":"00fc89611a59a444f43ff7e24d4fa0d9921a9594f0795024c758dbf8ae9aabb8"}
Oct 30 13:03:02.970687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.970637    4429 scope.go:122] "RemoveContainer" containerID="db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3"
Oct 30 13:03:02.976211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.976184    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-plugins-config-volume\") pod \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\" (UID: \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\") "
Oct 30 13:03:02.976555 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.976457    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-kube-api-access-8w5kn\" (UniqueName: \"kubernetes.io/projected/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-kube-api-access-8w5kn\") pod \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\" (UID: \"8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2\") "
Oct 30 13:03:02.976634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.976614    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-plugins-config-volume" pod "8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2" (UID: "8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:02.976943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.976925    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:02.979088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.979051    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-kube-api-access-8w5kn" pod "8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2" (UID: "8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2"). InnerVolumeSpecName "kube-api-access-8w5kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:02.982605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.982583    4429 scope.go:122] "RemoveContainer" containerID="db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3"
Oct 30 13:03:02.982937 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:02.982919    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3\": container with ID starting with db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3 not found: ID does not exist" containerID="db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3"
Oct 30 13:03:02.983008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:02.982945    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3"} err="failed to get container status \"db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3\": rpc error: code = NotFound desc = could not find container \"db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3\": container with ID starting with db5f41d632a82c1f9a64dc9af0f785ab210e66b579d6ca867247ba283b4b12d3 not found: ID does not exist"
Oct 30 13:03:03.078133 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:03.078102    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w5kn\" (UniqueName: \"kubernetes.io/projected/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2-kube-api-access-8w5kn\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:03.322051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:03.322004    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"]
Oct 30 13:03:03.328219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:03.328189    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ea117f29-85c756767f-lwk4k"]
Oct 30 13:03:04.076488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:04.076450    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2" path="/var/lib/kubelet/pods/8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2/volumes"
Oct 30 13:03:04.872610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:04.872585    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:03:04.872785 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:04.872657    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:03:06.767165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.767129    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"]
Oct 30 13:03:06.767820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.767805    4429 cpu_manager.go:414] "containerMap: removing container" podUID="8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2" containerName="payload-pre-processing"
Oct 30 13:03:06.767874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.767822    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2" containerName="payload-pre-processing"
Oct 30 13:03:06.767963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.767954    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d04efdc-d379-4b14-b9e6-7ac4ec47d0c2" containerName="payload-pre-processing"
Oct 30 13:03:06.772729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.772713    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:06.790256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.790230    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6f0f50\""
Oct 30 13:03:06.809045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.809013    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"]
Oct 30 13:03:06.853316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.853215    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092a541-4abc-40f0-9baa-e5af477e9972-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt\" (UID: \"b092a541-4abc-40f0-9baa-e5af477e9972\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:06.853504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.853433    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxpx\" (UniqueName: \"kubernetes.io/projected/b092a541-4abc-40f0-9baa-e5af477e9972-kube-api-access-mkxpx\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt\" (UID: \"b092a541-4abc-40f0-9baa-e5af477e9972\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:06.865454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.865421    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"]
Oct 30 13:03:06.869903 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.869879    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:06.888446 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.888411    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6f0f50-dockercfg-4l72k\""
Oct 30 13:03:06.889256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.889233    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"]
Oct 30 13:03:06.969324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.963223    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxpx\" (UniqueName: \"kubernetes.io/projected/b092a541-4abc-40f0-9baa-e5af477e9972-kube-api-access-mkxpx\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt\" (UID: \"b092a541-4abc-40f0-9baa-e5af477e9972\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:06.969324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.963348    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn\" (UID: \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:06.969324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.963601    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdlv\" (UniqueName: \"kubernetes.io/projected/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-kube-api-access-ckdlv\") pod \"payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn\" (UID: \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:06.969324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.963696    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092a541-4abc-40f0-9baa-e5af477e9972-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt\" (UID: \"b092a541-4abc-40f0-9baa-e5af477e9972\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:06.969324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:06.964784    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092a541-4abc-40f0-9baa-e5af477e9972-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt\" (UID: \"b092a541-4abc-40f0-9baa-e5af477e9972\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:07.002055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.002019    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxpx\" (UniqueName: \"kubernetes.io/projected/b092a541-4abc-40f0-9baa-e5af477e9972-kube-api-access-mkxpx\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt\" (UID: \"b092a541-4abc-40f0-9baa-e5af477e9972\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:07.070725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.070632    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn\" (UID: \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:07.070896 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.070833    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdlv\" (UniqueName: \"kubernetes.io/projected/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-kube-api-access-ckdlv\") pod \"payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn\" (UID: \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:07.072948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.072925    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn\" (UID: \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:07.087497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.087466    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:07.131911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.130050    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdlv\" (UniqueName: \"kubernetes.io/projected/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-kube-api-access-ckdlv\") pod \"payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn\" (UID: \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:07.184818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.184786    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:07.433117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.433094    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"]
Oct 30 13:03:07.436234 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:03:07.436204    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb092a541_4abc_40f0_9baa_e5af477e9972.slice/crio-d645a4d00e7920fec1803d14e1e1b8663fc4551110c278ee0cee300cf97732e2 WatchSource:0}: Error finding container d645a4d00e7920fec1803d14e1e1b8663fc4551110c278ee0cee300cf97732e2: Status 404 returned error can't find the container with id d645a4d00e7920fec1803d14e1e1b8663fc4551110c278ee0cee300cf97732e2
Oct 30 13:03:07.523660 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:03:07.523630    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cec1f6_77de_47ae_8ad0_a705e13e7d1c.slice/crio-7c53851f77c058c0e2cdf0bad8b6c03ea923927d14514433b190661d0a74efed WatchSource:0}: Error finding container 7c53851f77c058c0e2cdf0bad8b6c03ea923927d14514433b190661d0a74efed: Status 404 returned error can't find the container with id 7c53851f77c058c0e2cdf0bad8b6c03ea923927d14514433b190661d0a74efed
Oct 30 13:03:07.524695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:07.524675    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"]
Oct 30 13:03:08.013275 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:08.013238    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt" event={"ID":"b092a541-4abc-40f0-9baa-e5af477e9972","Type":"ContainerStarted","Data":"364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236"}
Oct 30 13:03:08.013739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:08.013319    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt" event={"ID":"b092a541-4abc-40f0-9baa-e5af477e9972","Type":"ContainerStarted","Data":"d645a4d00e7920fec1803d14e1e1b8663fc4551110c278ee0cee300cf97732e2"}
Oct 30 13:03:08.013739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:08.013462    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:08.014972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:08.014950    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn" event={"ID":"73cec1f6-77de-47ae-8ad0-a705e13e7d1c","Type":"ContainerStarted","Data":"194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07"}
Oct 30 13:03:08.015066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:08.014987    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn" event={"ID":"73cec1f6-77de-47ae-8ad0-a705e13e7d1c","Type":"ContainerStarted","Data":"7c53851f77c058c0e2cdf0bad8b6c03ea923927d14514433b190661d0a74efed"}
Oct 30 13:03:08.015066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:08.015041    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:08.064086 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:08.064037    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt" podStartSLOduration=2.064025434 podStartE2EDuration="2.064025434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:08.063789445 +0000 UTC m=+1846.119283270" watchObservedRunningTime="2026-10-30 13:03:08.064025434 +0000 UTC m=+1846.119519257"
Oct 30 13:03:08.109556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:08.109503    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn" podStartSLOduration=2.109489052 podStartE2EDuration="2.109489052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:08.108592781 +0000 UTC m=+1846.164086604" watchObservedRunningTime="2026-10-30 13:03:08.109489052 +0000 UTC m=+1846.164982876"
Oct 30 13:03:12.372826 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:12.372789    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:03:19.024819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:19.024782    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:19.025366 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:19.024864    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:26.383123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:26.383078    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"]
Oct 30 13:03:26.444433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:26.444397    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"]
Oct 30 13:03:27.267161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:27.267128    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"]
Oct 30 13:03:27.324498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:27.324468    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"]
Oct 30 13:03:31.383495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.383450    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml" podUID="ed4790f8-c8b3-4e43-a69a-0234bd9c3404" containerName="payload-processing" containerID="cri-o://d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669" gracePeriod=25
Oct 30 13:03:31.445125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.445084    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d" podUID="b8812311-393f-47a6-b825-ea8c144e320f" containerName="payload-pre-processing" containerID="cri-o://0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748" gracePeriod=25
Oct 30 13:03:31.741987 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.741962    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:03:31.763719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.763698    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:03:31.844869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.844835    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-kube-api-access-j9n9q\") pod \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\" (UID: \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\") "
Oct 30 13:03:31.845058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.845025    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-plugins-config-volume\") pod \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\" (UID: \"ed4790f8-c8b3-4e43-a69a-0234bd9c3404\") "
Oct 30 13:03:31.845624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.845592    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-plugins-config-volume" pod "ed4790f8-c8b3-4e43-a69a-0234bd9c3404" (UID: "ed4790f8-c8b3-4e43-a69a-0234bd9c3404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:31.845856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.845827    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:31.847493 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.847463    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-kube-api-access-j9n9q" pod "ed4790f8-c8b3-4e43-a69a-0234bd9c3404" (UID: "ed4790f8-c8b3-4e43-a69a-0234bd9c3404"). InnerVolumeSpecName "kube-api-access-j9n9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:31.946524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.946491    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8812311-393f-47a6-b825-ea8c144e320f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8812311-393f-47a6-b825-ea8c144e320f-plugins-config-volume\") pod \"b8812311-393f-47a6-b825-ea8c144e320f\" (UID: \"b8812311-393f-47a6-b825-ea8c144e320f\") "
Oct 30 13:03:31.946762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.946749    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8812311-393f-47a6-b825-ea8c144e320f-kube-api-access-6dm55\" (UniqueName: \"kubernetes.io/projected/b8812311-393f-47a6-b825-ea8c144e320f-kube-api-access-6dm55\") pod \"b8812311-393f-47a6-b825-ea8c144e320f\" (UID: \"b8812311-393f-47a6-b825-ea8c144e320f\") "
Oct 30 13:03:31.946908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.946883    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8812311-393f-47a6-b825-ea8c144e320f-plugins-config-volume" pod "b8812311-393f-47a6-b825-ea8c144e320f" (UID: "b8812311-393f-47a6-b825-ea8c144e320f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:31.947195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.947171    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8812311-393f-47a6-b825-ea8c144e320f-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:31.947341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.947199    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9n9q\" (UniqueName: \"kubernetes.io/projected/ed4790f8-c8b3-4e43-a69a-0234bd9c3404-kube-api-access-j9n9q\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:31.950142 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:31.950112    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8812311-393f-47a6-b825-ea8c144e320f-kube-api-access-6dm55" pod "b8812311-393f-47a6-b825-ea8c144e320f" (UID: "b8812311-393f-47a6-b825-ea8c144e320f"). InnerVolumeSpecName "kube-api-access-6dm55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:32.049107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.049005    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dm55\" (UniqueName: \"kubernetes.io/projected/b8812311-393f-47a6-b825-ea8c144e320f-kube-api-access-6dm55\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:32.207007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.206970    4429 generic.go:356] "Generic (PLEG): container finished" podID="b8812311-393f-47a6-b825-ea8c144e320f" containerID="0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748" exitCode=0
Oct 30 13:03:32.207170 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.207033    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"
Oct 30 13:03:32.207170 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.207048    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d" event={"ID":"b8812311-393f-47a6-b825-ea8c144e320f","Type":"ContainerDied","Data":"0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748"}
Oct 30 13:03:32.207170 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.207099    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d" event={"ID":"b8812311-393f-47a6-b825-ea8c144e320f","Type":"ContainerDied","Data":"44317821955c62914903861bd8683bc904ea499ecb9130e2981cb2db5a4e2a08"}
Oct 30 13:03:32.207170 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.207126    4429 scope.go:122] "RemoveContainer" containerID="0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748"
Oct 30 13:03:32.208756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.208731    4429 generic.go:356] "Generic (PLEG): container finished" podID="ed4790f8-c8b3-4e43-a69a-0234bd9c3404" containerID="d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669" exitCode=0
Oct 30 13:03:32.208899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.208794    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"
Oct 30 13:03:32.208899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.208807    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml" event={"ID":"ed4790f8-c8b3-4e43-a69a-0234bd9c3404","Type":"ContainerDied","Data":"d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669"}
Oct 30 13:03:32.208899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.208852    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml" event={"ID":"ed4790f8-c8b3-4e43-a69a-0234bd9c3404","Type":"ContainerDied","Data":"0a57547067bc2bee9e798d7e017d486595fcf42776a786cc05d083b74963b493"}
Oct 30 13:03:32.217132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.217115    4429 scope.go:122] "RemoveContainer" containerID="0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748"
Oct 30 13:03:32.217450 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:32.217432    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748\": container with ID starting with 0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748 not found: ID does not exist" containerID="0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748"
Oct 30 13:03:32.217510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.217457    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748"} err="failed to get container status \"0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748\": rpc error: code = NotFound desc = could not find container \"0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748\": container with ID starting with 0de1bd70ac99e5f215590ed55d9b538ba679add0c5ed5917dc14a3fb9d7ff748 not found: ID does not exist"
Oct 30 13:03:32.217510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.217477    4429 scope.go:122] "RemoveContainer" containerID="d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669"
Oct 30 13:03:32.226199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.226179    4429 scope.go:122] "RemoveContainer" containerID="d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669"
Oct 30 13:03:32.226480 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:32.226461    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669\": container with ID starting with d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669 not found: ID does not exist" containerID="d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669"
Oct 30 13:03:32.226528 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.226487    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669"} err="failed to get container status \"d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669\": rpc error: code = NotFound desc = could not find container \"d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669\": container with ID starting with d3da9bf307b9d3ce60c118d9a90002d03280f2782dcfb085c3fdafc5fd552669 not found: ID does not exist"
Oct 30 13:03:32.267457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.267426    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t" podUID="c18f2d58-52ea-405d-88a0-c7675dfdb3ea" containerName="payload-processing" containerID="cri-o://1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818" gracePeriod=25
Oct 30 13:03:32.271346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.271321    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"]
Oct 30 13:03:32.285603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.285583    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-79658d9d7b-76tml"]
Oct 30 13:03:32.313530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.313349    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"]
Oct 30 13:03:32.317884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.317862    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-5f45c9f579-znb9d"]
Oct 30 13:03:32.325944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.325911    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr" podUID="c56700fe-e3f5-41ed-90eb-03d636efe96b" containerName="payload-pre-processing" containerID="cri-o://f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922" gracePeriod=25
Oct 30 13:03:32.612707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.612683    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:03:32.645366 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.645344    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:03:32.760496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.760458    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56700fe-e3f5-41ed-90eb-03d636efe96b-kube-api-access-cs856\" (UniqueName: \"kubernetes.io/projected/c56700fe-e3f5-41ed-90eb-03d636efe96b-kube-api-access-cs856\") pod \"c56700fe-e3f5-41ed-90eb-03d636efe96b\" (UID: \"c56700fe-e3f5-41ed-90eb-03d636efe96b\") "
Oct 30 13:03:32.760719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.760630    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-kube-api-access-7gcw9\" (UniqueName: \"kubernetes.io/projected/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-kube-api-access-7gcw9\") pod \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\" (UID: \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\") "
Oct 30 13:03:32.760791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.760747    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56700fe-e3f5-41ed-90eb-03d636efe96b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56700fe-e3f5-41ed-90eb-03d636efe96b-plugins-config-volume\") pod \"c56700fe-e3f5-41ed-90eb-03d636efe96b\" (UID: \"c56700fe-e3f5-41ed-90eb-03d636efe96b\") "
Oct 30 13:03:32.760880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.760866    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-plugins-config-volume\") pod \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\" (UID: \"c18f2d58-52ea-405d-88a0-c7675dfdb3ea\") "
Oct 30 13:03:32.761181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.761156    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-plugins-config-volume" pod "c18f2d58-52ea-405d-88a0-c7675dfdb3ea" (UID: "c18f2d58-52ea-405d-88a0-c7675dfdb3ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:32.761387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.761364    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56700fe-e3f5-41ed-90eb-03d636efe96b-plugins-config-volume" pod "c56700fe-e3f5-41ed-90eb-03d636efe96b" (UID: "c56700fe-e3f5-41ed-90eb-03d636efe96b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:32.761556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.761538    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56700fe-e3f5-41ed-90eb-03d636efe96b-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:32.761601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.761564    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:32.762901 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.762882    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-kube-api-access-7gcw9" pod "c18f2d58-52ea-405d-88a0-c7675dfdb3ea" (UID: "c18f2d58-52ea-405d-88a0-c7675dfdb3ea"). InnerVolumeSpecName "kube-api-access-7gcw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:32.762965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.762915    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56700fe-e3f5-41ed-90eb-03d636efe96b-kube-api-access-cs856" pod "c56700fe-e3f5-41ed-90eb-03d636efe96b" (UID: "c56700fe-e3f5-41ed-90eb-03d636efe96b"). InnerVolumeSpecName "kube-api-access-cs856". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:32.862772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.862683    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs856\" (UniqueName: \"kubernetes.io/projected/c56700fe-e3f5-41ed-90eb-03d636efe96b-kube-api-access-cs856\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:32.862772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:32.862713    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcw9\" (UniqueName: \"kubernetes.io/projected/c18f2d58-52ea-405d-88a0-c7675dfdb3ea-kube-api-access-7gcw9\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:33.216630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.216593    4429 generic.go:356] "Generic (PLEG): container finished" podID="c18f2d58-52ea-405d-88a0-c7675dfdb3ea" containerID="1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818" exitCode=0
Oct 30 13:03:33.216814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.216648    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"
Oct 30 13:03:33.216814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.216676    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t" event={"ID":"c18f2d58-52ea-405d-88a0-c7675dfdb3ea","Type":"ContainerDied","Data":"1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818"}
Oct 30 13:03:33.216814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.216735    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t" event={"ID":"c18f2d58-52ea-405d-88a0-c7675dfdb3ea","Type":"ContainerDied","Data":"b2cf891f240c6d74f15d24f9bf8d0109feb428d261b4f68eb2ed643317c3b602"}
Oct 30 13:03:33.216814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.216753    4429 scope.go:122] "RemoveContainer" containerID="1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818"
Oct 30 13:03:33.218050 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.218020    4429 generic.go:356] "Generic (PLEG): container finished" podID="c56700fe-e3f5-41ed-90eb-03d636efe96b" containerID="f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922" exitCode=0
Oct 30 13:03:33.218179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.218065    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"
Oct 30 13:03:33.218179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.218109    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr" event={"ID":"c56700fe-e3f5-41ed-90eb-03d636efe96b","Type":"ContainerDied","Data":"f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922"}
Oct 30 13:03:33.218179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.218161    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr" event={"ID":"c56700fe-e3f5-41ed-90eb-03d636efe96b","Type":"ContainerDied","Data":"62373acd0b59fed4cae4b5b132068b1924d8a92dd0e18c96ee5f0a2f7ab814fd"}
Oct 30 13:03:33.228453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.228370    4429 scope.go:122] "RemoveContainer" containerID="1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818"
Oct 30 13:03:33.228682 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:33.228655    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818\": container with ID starting with 1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818 not found: ID does not exist" containerID="1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818"
Oct 30 13:03:33.228771 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.228692    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818"} err="failed to get container status \"1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818\": rpc error: code = NotFound desc = could not find container \"1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818\": container with ID starting with 1cc140ba873026965d52bf35509ce91d51dd20d1edd03c016202dc71d65e2818 not found: ID does not exist"
Oct 30 13:03:33.228771 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.228710    4429 scope.go:122] "RemoveContainer" containerID="f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922"
Oct 30 13:03:33.238108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.238092    4429 scope.go:122] "RemoveContainer" containerID="f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922"
Oct 30 13:03:33.238393 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:33.238374    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922\": container with ID starting with f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922 not found: ID does not exist" containerID="f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922"
Oct 30 13:03:33.238470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.238399    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922"} err="failed to get container status \"f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922\": rpc error: code = NotFound desc = could not find container \"f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922\": container with ID starting with f5c0c1d758681a82d55ff6d6f2ae06073aecdd99e2479d2ff445dd1f2efcf922 not found: ID does not exist"
Oct 30 13:03:33.275332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.275303    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"]
Oct 30 13:03:33.288971 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.288948    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f60f16-57546d5647-z6kxr"]
Oct 30 13:03:33.315418 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.315387    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"]
Oct 30 13:03:33.321767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:33.321731    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f60f16-6d7888d9c6-smv2t"]
Oct 30 13:03:34.078703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:34.078673    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8812311-393f-47a6-b825-ea8c144e320f" path="/var/lib/kubelet/pods/b8812311-393f-47a6-b825-ea8c144e320f/volumes"
Oct 30 13:03:34.079200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:34.079109    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18f2d58-52ea-405d-88a0-c7675dfdb3ea" path="/var/lib/kubelet/pods/c18f2d58-52ea-405d-88a0-c7675dfdb3ea/volumes"
Oct 30 13:03:34.079497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:34.079482    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56700fe-e3f5-41ed-90eb-03d636efe96b" path="/var/lib/kubelet/pods/c56700fe-e3f5-41ed-90eb-03d636efe96b/volumes"
Oct 30 13:03:34.079835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:34.079823    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4790f8-c8b3-4e43-a69a-0234bd9c3404" path="/var/lib/kubelet/pods/ed4790f8-c8b3-4e43-a69a-0234bd9c3404/volumes"
Oct 30 13:03:38.167193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.167155    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"]
Oct 30 13:03:38.168139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168112    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b8812311-393f-47a6-b825-ea8c144e320f" containerName="payload-pre-processing"
Oct 30 13:03:38.168139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168135    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8812311-393f-47a6-b825-ea8c144e320f" containerName="payload-pre-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168171    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c18f2d58-52ea-405d-88a0-c7675dfdb3ea" containerName="payload-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168176    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18f2d58-52ea-405d-88a0-c7675dfdb3ea" containerName="payload-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168203    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ed4790f8-c8b3-4e43-a69a-0234bd9c3404" containerName="payload-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168209    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4790f8-c8b3-4e43-a69a-0234bd9c3404" containerName="payload-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168216    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c56700fe-e3f5-41ed-90eb-03d636efe96b" containerName="payload-pre-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168221    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56700fe-e3f5-41ed-90eb-03d636efe96b" containerName="payload-pre-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168334    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18f2d58-52ea-405d-88a0-c7675dfdb3ea" containerName="payload-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168348    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4790f8-c8b3-4e43-a69a-0234bd9c3404" containerName="payload-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168357    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56700fe-e3f5-41ed-90eb-03d636efe96b" containerName="payload-pre-processing"
Oct 30 13:03:38.168368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.168368    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8812311-393f-47a6-b825-ea8c144e320f" containerName="payload-pre-processing"
Oct 30 13:03:38.173768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.173744    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:38.198222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.198189    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"]
Oct 30 13:03:38.207319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.206777    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9368ce\""
Oct 30 13:03:38.207319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.206828    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9368ce-dockercfg-2kw92\""
Oct 30 13:03:38.254194 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.254159    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c084f-7798-48c9-91b6-b0c26e39c49b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww\" (UID: \"522c084f-7798-48c9-91b6-b0c26e39c49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:38.254436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.254405    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsvp2\" (UniqueName: \"kubernetes.io/projected/522c084f-7798-48c9-91b6-b0c26e39c49b-kube-api-access-fsvp2\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww\" (UID: \"522c084f-7798-48c9-91b6-b0c26e39c49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:38.324359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.324327    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"]
Oct 30 13:03:38.335224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.335195    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:38.351390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.351342    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"]
Oct 30 13:03:38.355711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.355676    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsvp2\" (UniqueName: \"kubernetes.io/projected/522c084f-7798-48c9-91b6-b0c26e39c49b-kube-api-access-fsvp2\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww\" (UID: \"522c084f-7798-48c9-91b6-b0c26e39c49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:38.355909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.355833    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzx6x\" (UniqueName: \"kubernetes.io/projected/670fae55-f8db-47ed-8b56-e9974b2215d8-kube-api-access-bzx6x\") pod \"payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt\" (UID: \"670fae55-f8db-47ed-8b56-e9974b2215d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:38.355990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.355915    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c084f-7798-48c9-91b6-b0c26e39c49b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww\" (UID: \"522c084f-7798-48c9-91b6-b0c26e39c49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:38.356064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.356014    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fae55-f8db-47ed-8b56-e9974b2215d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt\" (UID: \"670fae55-f8db-47ed-8b56-e9974b2215d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:38.357189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.357162    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c084f-7798-48c9-91b6-b0c26e39c49b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww\" (UID: \"522c084f-7798-48c9-91b6-b0c26e39c49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:38.384217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.384190    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsvp2\" (UniqueName: \"kubernetes.io/projected/522c084f-7798-48c9-91b6-b0c26e39c49b-kube-api-access-fsvp2\") pod \"payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww\" (UID: \"522c084f-7798-48c9-91b6-b0c26e39c49b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:38.457626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.457601    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzx6x\" (UniqueName: \"kubernetes.io/projected/670fae55-f8db-47ed-8b56-e9974b2215d8-kube-api-access-bzx6x\") pod \"payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt\" (UID: \"670fae55-f8db-47ed-8b56-e9974b2215d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:38.457781 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.457710    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fae55-f8db-47ed-8b56-e9974b2215d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt\" (UID: \"670fae55-f8db-47ed-8b56-e9974b2215d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:38.458360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.458341    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fae55-f8db-47ed-8b56-e9974b2215d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt\" (UID: \"670fae55-f8db-47ed-8b56-e9974b2215d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:38.482800 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.482774    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzx6x\" (UniqueName: \"kubernetes.io/projected/670fae55-f8db-47ed-8b56-e9974b2215d8-kube-api-access-bzx6x\") pod \"payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt\" (UID: \"670fae55-f8db-47ed-8b56-e9974b2215d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:38.490304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.490259    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:38.690832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.690801    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:38.799123 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:03:38.799095    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522c084f_7798_48c9_91b6_b0c26e39c49b.slice/crio-2c9aed291e52292b984fb824ce0310c08c45bc0a797fc7fbf300d4fcc4cd5d2d WatchSource:0}: Error finding container 2c9aed291e52292b984fb824ce0310c08c45bc0a797fc7fbf300d4fcc4cd5d2d: Status 404 returned error can't find the container with id 2c9aed291e52292b984fb824ce0310c08c45bc0a797fc7fbf300d4fcc4cd5d2d
Oct 30 13:03:38.799123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:38.799106    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"]
Oct 30 13:03:39.006548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:39.006515    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"]
Oct 30 13:03:39.007321 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:03:39.007254    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670fae55_f8db_47ed_8b56_e9974b2215d8.slice/crio-bded35a5b5178057e15bcdc49a6a66e807288a26103d4c6a8e388865dbcb9cb6 WatchSource:0}: Error finding container bded35a5b5178057e15bcdc49a6a66e807288a26103d4c6a8e388865dbcb9cb6: Status 404 returned error can't find the container with id bded35a5b5178057e15bcdc49a6a66e807288a26103d4c6a8e388865dbcb9cb6
Oct 30 13:03:39.291715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:39.291626    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww" event={"ID":"522c084f-7798-48c9-91b6-b0c26e39c49b","Type":"ContainerStarted","Data":"2c9aed291e52292b984fb824ce0310c08c45bc0a797fc7fbf300d4fcc4cd5d2d"}
Oct 30 13:03:39.293165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:39.293140    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt" event={"ID":"670fae55-f8db-47ed-8b56-e9974b2215d8","Type":"ContainerStarted","Data":"bded35a5b5178057e15bcdc49a6a66e807288a26103d4c6a8e388865dbcb9cb6"}
Oct 30 13:03:40.285320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:40.285256    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"]
Oct 30 13:03:40.319966 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:40.319935    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"]
Oct 30 13:03:41.325336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:41.319700    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww" event={"ID":"522c084f-7798-48c9-91b6-b0c26e39c49b","Type":"ContainerStarted","Data":"b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496"}
Oct 30 13:03:41.325336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:41.319793    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:41.325336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:41.322313    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt" event={"ID":"670fae55-f8db-47ed-8b56-e9974b2215d8","Type":"ContainerStarted","Data":"0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea"}
Oct 30 13:03:41.325336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:41.323108    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:41.378632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:41.378569    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww" podStartSLOduration=1.350783504 podStartE2EDuration="3.378551217s" totalImagesPullingTime="2.027767713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:41.374218312 +0000 UTC m=+1879.429712156" watchObservedRunningTime="2026-10-30 13:03:41.378551217 +0000 UTC m=+1879.434045062"
Oct 30 13:03:41.434783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:41.434731    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt" podStartSLOduration=1.604074261 podStartE2EDuration="3.434715347s" totalImagesPullingTime="1.830641086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:41.430736695 +0000 UTC m=+1879.486230520" watchObservedRunningTime="2026-10-30 13:03:41.434715347 +0000 UTC m=+1879.490209171"
Oct 30 13:03:42.376882 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:42.376842    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:03:45.291357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.290430    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn" podUID="73cec1f6-77de-47ae-8ad0-a705e13e7d1c" containerName="payload-processing" containerID="cri-o://194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07" gracePeriod=25
Oct 30 13:03:45.320964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.320929    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt" podUID="b092a541-4abc-40f0-9baa-e5af477e9972" containerName="payload-pre-processing" containerID="cri-o://364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236" gracePeriod=25
Oct 30 13:03:45.609843 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.609819    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:45.636485 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.636464    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:45.671867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.671836    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-kube-api-access-ckdlv\" (UniqueName: \"kubernetes.io/projected/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-kube-api-access-ckdlv\") pod \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\" (UID: \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\") "
Oct 30 13:03:45.672023 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.671907    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-plugins-config-volume\") pod \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\" (UID: \"73cec1f6-77de-47ae-8ad0-a705e13e7d1c\") "
Oct 30 13:03:45.672983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.672954    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-plugins-config-volume" pod "73cec1f6-77de-47ae-8ad0-a705e13e7d1c" (UID: "73cec1f6-77de-47ae-8ad0-a705e13e7d1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:45.675061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.675032    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-kube-api-access-ckdlv" pod "73cec1f6-77de-47ae-8ad0-a705e13e7d1c" (UID: "73cec1f6-77de-47ae-8ad0-a705e13e7d1c"). InnerVolumeSpecName "kube-api-access-ckdlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:45.774015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.773984    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b092a541-4abc-40f0-9baa-e5af477e9972-kube-api-access-mkxpx\" (UniqueName: \"kubernetes.io/projected/b092a541-4abc-40f0-9baa-e5af477e9972-kube-api-access-mkxpx\") pod \"b092a541-4abc-40f0-9baa-e5af477e9972\" (UID: \"b092a541-4abc-40f0-9baa-e5af477e9972\") "
Oct 30 13:03:45.774161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.774108    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b092a541-4abc-40f0-9baa-e5af477e9972-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092a541-4abc-40f0-9baa-e5af477e9972-plugins-config-volume\") pod \"b092a541-4abc-40f0-9baa-e5af477e9972\" (UID: \"b092a541-4abc-40f0-9baa-e5af477e9972\") "
Oct 30 13:03:45.774524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.774494    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b092a541-4abc-40f0-9baa-e5af477e9972-plugins-config-volume" pod "b092a541-4abc-40f0-9baa-e5af477e9972" (UID: "b092a541-4abc-40f0-9baa-e5af477e9972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:45.774524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.774514    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdlv\" (UniqueName: \"kubernetes.io/projected/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-kube-api-access-ckdlv\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:45.774524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.774526    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73cec1f6-77de-47ae-8ad0-a705e13e7d1c-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:45.776414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.776390    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b092a541-4abc-40f0-9baa-e5af477e9972-kube-api-access-mkxpx" pod "b092a541-4abc-40f0-9baa-e5af477e9972" (UID: "b092a541-4abc-40f0-9baa-e5af477e9972"). InnerVolumeSpecName "kube-api-access-mkxpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:45.875761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.875672    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkxpx\" (UniqueName: \"kubernetes.io/projected/b092a541-4abc-40f0-9baa-e5af477e9972-kube-api-access-mkxpx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:45.875761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:45.875705    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b092a541-4abc-40f0-9baa-e5af477e9972-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:46.355708 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.355675    4429 generic.go:356] "Generic (PLEG): container finished" podID="b092a541-4abc-40f0-9baa-e5af477e9972" containerID="364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236" exitCode=0
Oct 30 13:03:46.356163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.355757    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"
Oct 30 13:03:46.356163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.355770    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt" event={"ID":"b092a541-4abc-40f0-9baa-e5af477e9972","Type":"ContainerDied","Data":"364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236"}
Oct 30 13:03:46.356163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.355852    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt" event={"ID":"b092a541-4abc-40f0-9baa-e5af477e9972","Type":"ContainerDied","Data":"d645a4d00e7920fec1803d14e1e1b8663fc4551110c278ee0cee300cf97732e2"}
Oct 30 13:03:46.356163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.355879    4429 scope.go:122] "RemoveContainer" containerID="364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236"
Oct 30 13:03:46.357336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.357278    4429 generic.go:356] "Generic (PLEG): container finished" podID="73cec1f6-77de-47ae-8ad0-a705e13e7d1c" containerID="194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07" exitCode=0
Oct 30 13:03:46.357486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.357466    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn" event={"ID":"73cec1f6-77de-47ae-8ad0-a705e13e7d1c","Type":"ContainerDied","Data":"194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07"}
Oct 30 13:03:46.357576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.357514    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn" event={"ID":"73cec1f6-77de-47ae-8ad0-a705e13e7d1c","Type":"ContainerDied","Data":"7c53851f77c058c0e2cdf0bad8b6c03ea923927d14514433b190661d0a74efed"}
Oct 30 13:03:46.357576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.357537    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"
Oct 30 13:03:46.367684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.367665    4429 scope.go:122] "RemoveContainer" containerID="364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236"
Oct 30 13:03:46.368028 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:46.367998    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236\": container with ID starting with 364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236 not found: ID does not exist" containerID="364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236"
Oct 30 13:03:46.368135 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.368039    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236"} err="failed to get container status \"364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236\": rpc error: code = NotFound desc = could not find container \"364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236\": container with ID starting with 364357d9270451b7be9f8fe9bb0f19c386543f2f496836c3abf7fc4b27b47236 not found: ID does not exist"
Oct 30 13:03:46.368135 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.368066    4429 scope.go:122] "RemoveContainer" containerID="194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07"
Oct 30 13:03:46.378017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.377990    4429 scope.go:122] "RemoveContainer" containerID="194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07"
Oct 30 13:03:46.378393 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:46.378374    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07\": container with ID starting with 194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07 not found: ID does not exist" containerID="194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07"
Oct 30 13:03:46.378458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.378400    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07"} err="failed to get container status \"194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07\": rpc error: code = NotFound desc = could not find container \"194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07\": container with ID starting with 194f782bf913baab3e59c339235e1eaf1a8eacdac6387c2fb08c17b7776bfd07 not found: ID does not exist"
Oct 30 13:03:46.413266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.413240    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"]
Oct 30 13:03:46.426698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.426668    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-58d7c5b446-6cwjn"]
Oct 30 13:03:46.452279 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.452245    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"]
Oct 30 13:03:46.455691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:46.455669    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-6cfbb89f67-dnjmt"]
Oct 30 13:03:47.936699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:47.936663    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"]
Oct 30 13:03:47.937145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:47.936890    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b" podUID="df478c39-3038-43d3-9dc3-14be3ba3f3a9" containerName="maas-api" containerID="cri-o://0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6" gracePeriod=30
Oct 30 13:03:48.075894 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.075861    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73cec1f6-77de-47ae-8ad0-a705e13e7d1c" path="/var/lib/kubelet/pods/73cec1f6-77de-47ae-8ad0-a705e13e7d1c/volumes"
Oct 30 13:03:48.076431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.076415    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b092a541-4abc-40f0-9baa-e5af477e9972" path="/var/lib/kubelet/pods/b092a541-4abc-40f0-9baa-e5af477e9972/volumes"
Oct 30 13:03:48.275138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.275116    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:03:48.378180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.378145    4429 generic.go:356] "Generic (PLEG): container finished" podID="df478c39-3038-43d3-9dc3-14be3ba3f3a9" containerID="0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6" exitCode=0
Oct 30 13:03:48.378404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.378352    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b" event={"ID":"df478c39-3038-43d3-9dc3-14be3ba3f3a9","Type":"ContainerDied","Data":"0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6"}
Oct 30 13:03:48.378527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.378406    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b" event={"ID":"df478c39-3038-43d3-9dc3-14be3ba3f3a9","Type":"ContainerDied","Data":"4a52baf2c811bd09051fa01ee66a5eaa7d3bdeed5e2aaa2af44f4ae33f2b2a17"}
Oct 30 13:03:48.378527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.378435    4429 scope.go:122] "RemoveContainer" containerID="0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6"
Oct 30 13:03:48.378639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.378550    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"
Oct 30 13:03:48.401083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.401062    4429 scope.go:122] "RemoveContainer" containerID="0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6"
Oct 30 13:03:48.401676 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:03:48.401638    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6\": container with ID starting with 0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6 not found: ID does not exist" containerID="0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6"
Oct 30 13:03:48.401795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.401686    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6"} err="failed to get container status \"0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6\": rpc error: code = NotFound desc = could not find container \"0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6\": container with ID starting with 0fe6f0b745cdead40d5e989965cd4409ba5fdf66fe2ba2d4887b2966b8fefdf6 not found: ID does not exist"
Oct 30 13:03:48.423935 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.423906    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df478c39-3038-43d3-9dc3-14be3ba3f3a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df478c39-3038-43d3-9dc3-14be3ba3f3a9-openshift-service-ca\") pod \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") "
Oct 30 13:03:48.424363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.424343    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df478c39-3038-43d3-9dc3-14be3ba3f3a9-kube-api-access-5292w\" (UniqueName: \"kubernetes.io/projected/df478c39-3038-43d3-9dc3-14be3ba3f3a9-kube-api-access-5292w\") pod \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") "
Oct 30 13:03:48.424604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.424574    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-maas-api-tls\") pod \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") "
Oct 30 13:03:48.424742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.424725    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-metrics-certs\") pod \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\" (UID: \"df478c39-3038-43d3-9dc3-14be3ba3f3a9\") "
Oct 30 13:03:48.426547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.426511    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df478c39-3038-43d3-9dc3-14be3ba3f3a9-openshift-service-ca" pod "df478c39-3038-43d3-9dc3-14be3ba3f3a9" (UID: "df478c39-3038-43d3-9dc3-14be3ba3f3a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:03:48.434544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.434511    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-metrics-certs" pod "df478c39-3038-43d3-9dc3-14be3ba3f3a9" (UID: "df478c39-3038-43d3-9dc3-14be3ba3f3a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:48.439798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.439739    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-maas-api-tls" pod "df478c39-3038-43d3-9dc3-14be3ba3f3a9" (UID: "df478c39-3038-43d3-9dc3-14be3ba3f3a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:03:48.441182 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.441159    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df478c39-3038-43d3-9dc3-14be3ba3f3a9-kube-api-access-5292w" pod "df478c39-3038-43d3-9dc3-14be3ba3f3a9" (UID: "df478c39-3038-43d3-9dc3-14be3ba3f3a9"). InnerVolumeSpecName "kube-api-access-5292w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:03:48.535413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.526764    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5292w\" (UniqueName: \"kubernetes.io/projected/df478c39-3038-43d3-9dc3-14be3ba3f3a9-kube-api-access-5292w\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:48.535413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.526803    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:48.535413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.526818    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df478c39-3038-43d3-9dc3-14be3ba3f3a9-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:48.535413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.526832    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df478c39-3038-43d3-9dc3-14be3ba3f3a9-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:03:48.741718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.741687    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"]
Oct 30 13:03:48.744540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:48.744520    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f60f16-98994945-wgz6b"]
Oct 30 13:03:50.075799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:50.075758    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df478c39-3038-43d3-9dc3-14be3ba3f3a9" path="/var/lib/kubelet/pods/df478c39-3038-43d3-9dc3-14be3ba3f3a9/volumes"
Oct 30 13:03:51.923122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.923084    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"]
Oct 30 13:03:51.924051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924030    4429 cpu_manager.go:414] "containerMap: removing container" podUID="df478c39-3038-43d3-9dc3-14be3ba3f3a9" containerName="maas-api"
Oct 30 13:03:51.924101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924055    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df478c39-3038-43d3-9dc3-14be3ba3f3a9" containerName="maas-api"
Oct 30 13:03:51.924143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924126    4429 cpu_manager.go:414] "containerMap: removing container" podUID="73cec1f6-77de-47ae-8ad0-a705e13e7d1c" containerName="payload-processing"
Oct 30 13:03:51.924143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924133    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73cec1f6-77de-47ae-8ad0-a705e13e7d1c" containerName="payload-processing"
Oct 30 13:03:51.924201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924151    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b092a541-4abc-40f0-9baa-e5af477e9972" containerName="payload-pre-processing"
Oct 30 13:03:51.924201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924159    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b092a541-4abc-40f0-9baa-e5af477e9972" containerName="payload-pre-processing"
Oct 30 13:03:51.924333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924321    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b092a541-4abc-40f0-9baa-e5af477e9972" containerName="payload-pre-processing"
Oct 30 13:03:51.924367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924336    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="df478c39-3038-43d3-9dc3-14be3ba3f3a9" containerName="maas-api"
Oct 30 13:03:51.924367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.924357    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="73cec1f6-77de-47ae-8ad0-a705e13e7d1c" containerName="payload-processing"
Oct 30 13:03:51.929985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.929960    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:51.953344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.953313    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"]
Oct 30 13:03:51.961585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.961556    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6f0f50\""
Oct 30 13:03:51.961767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:51.961627    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6f0f50-dockercfg-qzrx9\""
Oct 30 13:03:52.079124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.079092    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0c4a8b-81f8-40c4-b23a-8bef204310da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n\" (UID: \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:52.080577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.080341    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rgs\" (UniqueName: \"kubernetes.io/projected/6f0c4a8b-81f8-40c4-b23a-8bef204310da-kube-api-access-w2rgs\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n\" (UID: \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:52.093118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.093087    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"]
Oct 30 13:03:52.097396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.097359    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:52.118474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.118438    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"]
Oct 30 13:03:52.181396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.181310    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdr2\" (UniqueName: \"kubernetes.io/projected/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-kube-api-access-kvdr2\") pod \"payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl\" (UID: \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:52.181396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.181350    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl\" (UID: \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:52.181396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.181381    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0c4a8b-81f8-40c4-b23a-8bef204310da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n\" (UID: \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:52.181921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.181896    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rgs\" (UniqueName: \"kubernetes.io/projected/6f0c4a8b-81f8-40c4-b23a-8bef204310da-kube-api-access-w2rgs\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n\" (UID: \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:52.182237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.182218    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0c4a8b-81f8-40c4-b23a-8bef204310da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n\" (UID: \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:52.210958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.210920    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rgs\" (UniqueName: \"kubernetes.io/projected/6f0c4a8b-81f8-40c4-b23a-8bef204310da-kube-api-access-w2rgs\") pod \"payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n\" (UID: \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:52.252582 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.252556    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:52.283031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.282996    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdr2\" (UniqueName: \"kubernetes.io/projected/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-kube-api-access-kvdr2\") pod \"payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl\" (UID: \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:52.283194 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.283051    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl\" (UID: \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:52.283934 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.283912    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl\" (UID: \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:52.312450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.312417    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdr2\" (UniqueName: \"kubernetes.io/projected/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-kube-api-access-kvdr2\") pod \"payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl\" (UID: \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:52.332479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.332095    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:03:52.412059 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.412024    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:52.565324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.565269    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"]
Oct 30 13:03:52.566977 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:03:52.566944    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0c4a8b_81f8_40c4_b23a_8bef204310da.slice/crio-b819d6a8384a45ad8e98e52c26a1579e85bd25aebfb5bc01536439319c692402 WatchSource:0}: Error finding container b819d6a8384a45ad8e98e52c26a1579e85bd25aebfb5bc01536439319c692402: Status 404 returned error can't find the container with id b819d6a8384a45ad8e98e52c26a1579e85bd25aebfb5bc01536439319c692402
Oct 30 13:03:52.727236 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:03:52.727205    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c41a54c_071c_4c7c_ac6d_0cee9dc36ee8.slice/crio-6d9294d2868b9410736f5d535523dc97e3437523f6242419d3d91703964b3abb WatchSource:0}: Error finding container 6d9294d2868b9410736f5d535523dc97e3437523f6242419d3d91703964b3abb: Status 404 returned error can't find the container with id 6d9294d2868b9410736f5d535523dc97e3437523f6242419d3d91703964b3abb
Oct 30 13:03:52.727573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:52.727500    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"]
Oct 30 13:03:53.339354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.339327    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:03:53.425525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.425042    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl" event={"ID":"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8","Type":"ContainerStarted","Data":"6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1"}
Oct 30 13:03:53.425741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.425574    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl" event={"ID":"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8","Type":"ContainerStarted","Data":"6d9294d2868b9410736f5d535523dc97e3437523f6242419d3d91703964b3abb"}
Oct 30 13:03:53.425741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.425617    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:03:53.427607 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.427582    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n" event={"ID":"6f0c4a8b-81f8-40c4-b23a-8bef204310da","Type":"ContainerStarted","Data":"498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650"}
Oct 30 13:03:53.427687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.427626    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n" event={"ID":"6f0c4a8b-81f8-40c4-b23a-8bef204310da","Type":"ContainerStarted","Data":"b819d6a8384a45ad8e98e52c26a1579e85bd25aebfb5bc01536439319c692402"}
Oct 30 13:03:53.427792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.427775    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:03:53.477051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.477003    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl" podStartSLOduration=1.476988818 podStartE2EDuration="1.476988818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:53.47371582 +0000 UTC m=+1891.529209641" watchObservedRunningTime="2026-10-30 13:03:53.476988818 +0000 UTC m=+1891.532482641"
Oct 30 13:03:53.523471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:53.523418    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n" podStartSLOduration=2.523403356 podStartE2EDuration="2.523403356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:53.521208166 +0000 UTC m=+1891.576701989" watchObservedRunningTime="2026-10-30 13:03:53.523403356 +0000 UTC m=+1891.578897180"
Oct 30 13:03:58.743204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.743168    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"]
Oct 30 13:03:58.747745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.747724    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.772913 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.772440    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"]
Oct 30 13:03:58.792153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.791598    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-405eb1-openshift-default-dockercfg-jmlgx\""
Oct 30 13:03:58.792458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.792433    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51866e4a-2e91-47b3-bd71-99ca894f27d9-istiod-ca-cert\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.792571 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.792468    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-podinfo\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.792653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.792628    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27xt\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-kube-api-access-k27xt\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.792719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.792686    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-certs\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.793164 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.793143    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-envoy\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.793375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.793358    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-socket\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.793497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.793480    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-data\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.793619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.793602    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-credential-socket\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.793707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.793647    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-token\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.896313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.895298    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27xt\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-kube-api-access-k27xt\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.896313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.895857    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-certs\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898182    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-envoy\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898246    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-socket\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898580    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-certs\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898724    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-socket\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898793    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-data\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898825    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-credential-socket\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898862    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-token\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898895    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51866e4a-2e91-47b3-bd71-99ca894f27d9-istiod-ca-cert\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.898922    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-podinfo\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.899580    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-credential-socket\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.900348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.899773    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-data\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.901008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.900577    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51866e4a-2e91-47b3-bd71-99ca894f27d9-istiod-ca-cert\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.901501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.901479    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-envoy\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.903138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.903113    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-podinfo\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.967545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.959045    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-token\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:58.967807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:58.960901    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27xt\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-kube-api-access-k27xt\") pod \"e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") " pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:59.073450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:59.073380    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:03:59.212898 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:59.212850    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"]
Oct 30 13:03:59.213218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:59.213146    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy" containerID="cri-o://5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918" gracePeriod=30
Oct 30 13:03:59.411420 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:03:59.411392    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51866e4a_2e91_47b3_bd71_99ca894f27d9.slice/crio-c5794fd4337be976c22ed1dc19631f6cdd937c0e340487fc6eb35209d0c08613 WatchSource:0}: Error finding container c5794fd4337be976c22ed1dc19631f6cdd937c0e340487fc6eb35209d0c08613: Status 404 returned error can't find the container with id c5794fd4337be976c22ed1dc19631f6cdd937c0e340487fc6eb35209d0c08613
Oct 30 13:03:59.412473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:59.412448    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"]
Oct 30 13:03:59.414117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:59.414089    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:59.414184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:59.414152    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:59.414221 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:59.414182    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:59.511019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:03:59.510885    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" event={"ID":"51866e4a-2e91-47b3-bd71-99ca894f27d9","Type":"ContainerStarted","Data":"c5794fd4337be976c22ed1dc19631f6cdd937c0e340487fc6eb35209d0c08613"}
Oct 30 13:04:00.521154 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:00.521119    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" event={"ID":"51866e4a-2e91-47b3-bd71-99ca894f27d9","Type":"ContainerStarted","Data":"0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852"}
Oct 30 13:04:00.602554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:00.602493    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" podStartSLOduration=2.602476761 podStartE2EDuration="2.602476761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:00.599610034 +0000 UTC m=+1898.655103891" watchObservedRunningTime="2026-10-30 13:04:00.602476761 +0000 UTC m=+1898.657970584"
Oct 30 13:04:01.074216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:01.074182    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:04:01.080025 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:01.079998    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:04:01.528432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:01.528392    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:04:01.529414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:01.529394    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:04:03.229916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:03.229870    4429 patch_prober.go:29] interesting pod/e2e-mt-ea117f29-openshift-default-7c448945-djmq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:04:03.230429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:03.229941    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:04:04.438550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.438378    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:04:04.440001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.439160    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:04:04.519210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.519184    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 13:04:04.552961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.552930    4429 generic.go:356] "Generic (PLEG): container finished" podID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerID="5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918" exitCode=0
Oct 30 13:04:04.553115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.552968    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" event={"ID":"ecd6158f-d857-4148-b91d-f67591bd37dd","Type":"ContainerDied","Data":"5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918"}
Oct 30 13:04:04.553115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.552982    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"
Oct 30 13:04:04.553115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.553005    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5" event={"ID":"ecd6158f-d857-4148-b91d-f67591bd37dd","Type":"ContainerDied","Data":"01792bcc3d630defca48d4258cf4ed0d0d074fb62b6760f2505f42c5539b35cb"}
Oct 30 13:04:04.553115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.553026    4429 scope.go:122] "RemoveContainer" containerID="5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918"
Oct 30 13:04:04.563320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.563299    4429 scope.go:122] "RemoveContainer" containerID="bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e"
Oct 30 13:04:04.572413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.572396    4429 scope.go:122] "RemoveContainer" containerID="5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918"
Oct 30 13:04:04.572687 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:04.572665    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918\": container with ID starting with 5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918 not found: ID does not exist" containerID="5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918"
Oct 30 13:04:04.572740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.572697    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918"} err="failed to get container status \"5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918\": rpc error: code = NotFound desc = could not find container \"5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918\": container with ID starting with 5ba25c556de60b151dae1e71a20ae9dcaa3f03803196d87f102517227a1ce918 not found: ID does not exist"
Oct 30 13:04:04.572740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.572717    4429 scope.go:122] "RemoveContainer" containerID="bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e"
Oct 30 13:04:04.572937 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:04.572921    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e\": container with ID starting with bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e not found: ID does not exist" containerID="bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e"
Oct 30 13:04:04.572975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.572943    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e"} err="failed to get container status \"bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e\": rpc error: code = NotFound desc = could not find container \"bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e\": container with ID starting with bfa1b8c8be982ba57f1c271366bcb9767ff6ca7cf26a8df525889259ce91808e not found: ID does not exist"
Oct 30 13:04:04.618208 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618172    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-envoy\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.618208 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618220    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd6158f-d857-4148-b91d-f67591bd37dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd6158f-d857-4148-b91d-f67591bd37dd-istiod-ca-cert\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.618534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618254    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-token\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-token\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.618534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618343    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-credential-socket\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.618534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618398    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-socket\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.618534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618481    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-podinfo\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.618534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618518    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-data\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.618918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618581    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-kube-api-access-ds9dk\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-kube-api-access-ds9dk\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.618918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618672    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd6158f-d857-4148-b91d-f67591bd37dd-istiod-ca-cert" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:04.618918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618820    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-certs\") pod \"ecd6158f-d857-4148-b91d-f67591bd37dd\" (UID: \"ecd6158f-d857-4148-b91d-f67591bd37dd\") "
Oct 30 13:04:04.619145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.618929    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-socket" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:04.619746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.619484    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd6158f-d857-4148-b91d-f67591bd37dd-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.619746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.619511    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.619746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.619603    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-data" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:04.621200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.621178    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-envoy" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:04.622022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.621835    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-certs" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:04.622212 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.622163    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-credential-socket" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:04.623689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.622768    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-token" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:04.623689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.623606    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-kube-api-access-ds9dk" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "kube-api-access-ds9dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:04.624145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.624122    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-podinfo" pod "ecd6158f-d857-4148-b91d-f67591bd37dd" (UID: "ecd6158f-d857-4148-b91d-f67591bd37dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:04:04.721096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.721062    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.721096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.721095    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.721096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.721103    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.721354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.721112    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.721354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.721121    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.721354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.721132    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd6158f-d857-4148-b91d-f67591bd37dd-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.721354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.721166    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9dk\" (UniqueName: \"kubernetes.io/projected/ecd6158f-d857-4148-b91d-f67591bd37dd-kube-api-access-ds9dk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:04.852586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.852502    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"]
Oct 30 13:04:04.853254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.853232    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy"
Oct 30 13:04:04.853254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.853254    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy"
Oct 30 13:04:04.853478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.853266    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy"
Oct 30 13:04:04.853478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.853274    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy"
Oct 30 13:04:04.853589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.853486    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy"
Oct 30 13:04:04.853589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.853500    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" containerName="istio-proxy"
Oct 30 13:04:04.859082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.859060    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:04.875833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.875806    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-405eb1\""
Oct 30 13:04:04.876784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.876572    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-405eb1-dockercfg-jxvt6\""
Oct 30 13:04:04.879010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:04.878985    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"]
Oct 30 13:04:05.066316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.057495    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"]
Oct 30 13:04:05.101350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.096491    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xthq8\" (UniqueName: \"kubernetes.io/projected/5948d55b-48cd-4a7f-a51e-c19a8c03740e-kube-api-access-xthq8\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh\" (UID: \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:05.101350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.096589    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5948d55b-48cd-4a7f-a51e-c19a8c03740e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh\" (UID: \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:05.117462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.117303    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ea117f29-openshift-default-7c448945-djmq5"]
Oct 30 13:04:05.198168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.198137    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthq8\" (UniqueName: \"kubernetes.io/projected/5948d55b-48cd-4a7f-a51e-c19a8c03740e-kube-api-access-xthq8\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh\" (UID: \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:05.198364 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.198187    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5948d55b-48cd-4a7f-a51e-c19a8c03740e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh\" (UID: \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:05.199155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.199126    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5948d55b-48cd-4a7f-a51e-c19a8c03740e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh\" (UID: \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:05.223154 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.223125    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthq8\" (UniqueName: \"kubernetes.io/projected/5948d55b-48cd-4a7f-a51e-c19a8c03740e-kube-api-access-xthq8\") pod \"payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh\" (UID: \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:05.475920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.475891    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:05.785441 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:04:05.785410    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5948d55b_48cd_4a7f_a51e_c19a8c03740e.slice/crio-5152ae285355a32a5b67a45a40f8fa1678dd704c7676e8bda21c17213fa8cd16 WatchSource:0}: Error finding container 5152ae285355a32a5b67a45a40f8fa1678dd704c7676e8bda21c17213fa8cd16: Status 404 returned error can't find the container with id 5152ae285355a32a5b67a45a40f8fa1678dd704c7676e8bda21c17213fa8cd16
Oct 30 13:04:05.786417 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:05.786347    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"]
Oct 30 13:04:06.076005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:06.075927    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd6158f-d857-4148-b91d-f67591bd37dd" path="/var/lib/kubelet/pods/ecd6158f-d857-4148-b91d-f67591bd37dd/volumes"
Oct 30 13:04:06.575467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:06.575428    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh" event={"ID":"5948d55b-48cd-4a7f-a51e-c19a8c03740e","Type":"ContainerStarted","Data":"ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a"}
Oct 30 13:04:06.575919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:06.575509    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:06.575919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:06.575527    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh" event={"ID":"5948d55b-48cd-4a7f-a51e-c19a8c03740e","Type":"ContainerStarted","Data":"5152ae285355a32a5b67a45a40f8fa1678dd704c7676e8bda21c17213fa8cd16"}
Oct 30 13:04:06.634992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:06.634940    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh" podStartSLOduration=2.634927078 podStartE2EDuration="2.634927078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:06.633084921 +0000 UTC m=+1904.688578746" watchObservedRunningTime="2026-10-30 13:04:06.634927078 +0000 UTC m=+1904.690420901"
Oct 30 13:04:07.183445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.183403    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"]
Oct 30 13:04:07.809592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.809557    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"]
Oct 30 13:04:07.815394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.815369    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.844764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.844726    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"]
Oct 30 13:04:07.850236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.850209    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a1ee32-openshift-default-dockercfg-kh75g\""
Oct 30 13:04:07.940784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.940729    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-data\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.941279 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.941219    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-socket\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.941616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.941580    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-podinfo\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.941746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.941661    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657cfe5f-d9ae-42da-8649-e7de001439cd-istiod-ca-cert\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.941812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.941786    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-token\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.942321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.941836    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4qq\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-kube-api-access-4w4qq\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.942321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.942007    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-certs\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.942321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.942090    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-envoy\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:07.942321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:07.942175    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-credential-socket\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.043785 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.043750    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-certs\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.043974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.043833    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-envoy\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.043974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.043893    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-credential-socket\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.043974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.043935    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-data\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.043988    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-socket\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044013    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-podinfo\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044041    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657cfe5f-d9ae-42da-8649-e7de001439cd-istiod-ca-cert\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044108    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-token\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044150    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4qq\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-kube-api-access-4w4qq\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044227    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-certs\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044346    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-socket\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044380    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-data\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044475    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-credential-socket\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.044963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.044937    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657cfe5f-d9ae-42da-8649-e7de001439cd-istiod-ca-cert\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.046915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.046890    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-envoy\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.046999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.046907    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-podinfo\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.081760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.081606    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-token\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.082026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.081806    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4qq\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-kube-api-access-4w4qq\") pod \"e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") " pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.147403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.147368    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:08.480177 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:04:08.480148    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657cfe5f_d9ae_42da_8649_e7de001439cd.slice/crio-17e937c9ccc40303f52b336f71e9283ceb422da7b182a44bbe4cef41dbe8a873 WatchSource:0}: Error finding container 17e937c9ccc40303f52b336f71e9283ceb422da7b182a44bbe4cef41dbe8a873: Status 404 returned error can't find the container with id 17e937c9ccc40303f52b336f71e9283ceb422da7b182a44bbe4cef41dbe8a873
Oct 30 13:04:08.481233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.481199    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"]
Oct 30 13:04:08.483063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.483028    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:08.483190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.483127    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:08.483190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.483174    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:08.592321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.592263    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" event={"ID":"657cfe5f-d9ae-42da-8649-e7de001439cd","Type":"ContainerStarted","Data":"85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86"}
Oct 30 13:04:08.592487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.592349    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" event={"ID":"657cfe5f-d9ae-42da-8649-e7de001439cd","Type":"ContainerStarted","Data":"17e937c9ccc40303f52b336f71e9283ceb422da7b182a44bbe4cef41dbe8a873"}
Oct 30 13:04:08.645560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:08.645510    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podStartSLOduration=1.645491411 podStartE2EDuration="1.645491411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:08.641641258 +0000 UTC m=+1906.697135082" watchObservedRunningTime="2026-10-30 13:04:08.645491411 +0000 UTC m=+1906.700985240"
Oct 30 13:04:09.148223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:09.148193    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:09.153517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:09.153496    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:09.598580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:09.598554    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:09.599411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:09.599392    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:04:12.184573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.184532    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn" podUID="ddffcdc2-3b27-4023-bd57-18e36871e8de" containerName="payload-processing" containerID="cri-o://5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7" gracePeriod=25
Oct 30 13:04:12.396463 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:12.396411    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:04:12.473788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.473763    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:04:12.607952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.607918    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddffcdc2-3b27-4023-bd57-18e36871e8de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddffcdc2-3b27-4023-bd57-18e36871e8de-plugins-config-volume\") pod \"ddffcdc2-3b27-4023-bd57-18e36871e8de\" (UID: \"ddffcdc2-3b27-4023-bd57-18e36871e8de\") "
Oct 30 13:04:12.608162 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.607967    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddffcdc2-3b27-4023-bd57-18e36871e8de-kube-api-access-7tqpd\" (UniqueName: \"kubernetes.io/projected/ddffcdc2-3b27-4023-bd57-18e36871e8de-kube-api-access-7tqpd\") pod \"ddffcdc2-3b27-4023-bd57-18e36871e8de\" (UID: \"ddffcdc2-3b27-4023-bd57-18e36871e8de\") "
Oct 30 13:04:12.608411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.608380    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddffcdc2-3b27-4023-bd57-18e36871e8de-plugins-config-volume" pod "ddffcdc2-3b27-4023-bd57-18e36871e8de" (UID: "ddffcdc2-3b27-4023-bd57-18e36871e8de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:12.608668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.608648    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddffcdc2-3b27-4023-bd57-18e36871e8de-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:12.610752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.610728    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddffcdc2-3b27-4023-bd57-18e36871e8de-kube-api-access-7tqpd" pod "ddffcdc2-3b27-4023-bd57-18e36871e8de" (UID: "ddffcdc2-3b27-4023-bd57-18e36871e8de"). InnerVolumeSpecName "kube-api-access-7tqpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:12.619692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.619646    4429 generic.go:356] "Generic (PLEG): container finished" podID="ddffcdc2-3b27-4023-bd57-18e36871e8de" containerID="5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7" exitCode=0
Oct 30 13:04:12.619807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.619738    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn" event={"ID":"ddffcdc2-3b27-4023-bd57-18e36871e8de","Type":"ContainerDied","Data":"5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7"}
Oct 30 13:04:12.619807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.619769    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"
Oct 30 13:04:12.619807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.619786    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn" event={"ID":"ddffcdc2-3b27-4023-bd57-18e36871e8de","Type":"ContainerDied","Data":"c7937ebc25df9ba4e9b0d58433ce3b34874160cf0e99a2c4153cfb2e6363f3f9"}
Oct 30 13:04:12.619962 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.619809    4429 scope.go:122] "RemoveContainer" containerID="5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7"
Oct 30 13:04:12.631078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.631062    4429 scope.go:122] "RemoveContainer" containerID="5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7"
Oct 30 13:04:12.631403 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:12.631379    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7\": container with ID starting with 5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7 not found: ID does not exist" containerID="5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7"
Oct 30 13:04:12.631479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.631411    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7"} err="failed to get container status \"5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7\": rpc error: code = NotFound desc = could not find container \"5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7\": container with ID starting with 5554c39f459615ac99f91693bbe78e795e8f3cd492effcc3fb6e39f12ea8f5e7 not found: ID does not exist"
Oct 30 13:04:12.678380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.678356    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"]
Oct 30 13:04:12.681505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.681488    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2768f-5b7fc6c6d8-cqfzn"]
Oct 30 13:04:12.710002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.709947    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tqpd\" (UniqueName: \"kubernetes.io/projected/ddffcdc2-3b27-4023-bd57-18e36871e8de-kube-api-access-7tqpd\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:12.974757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.974678    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"]
Oct 30 13:04:12.975319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.975302    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ddffcdc2-3b27-4023-bd57-18e36871e8de" containerName="payload-processing"
Oct 30 13:04:12.975319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.975317    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddffcdc2-3b27-4023-bd57-18e36871e8de" containerName="payload-processing"
Oct 30 13:04:12.975503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.975447    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddffcdc2-3b27-4023-bd57-18e36871e8de" containerName="payload-processing"
Oct 30 13:04:12.981630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:12.981608    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:13.002930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.002904    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a1ee32-dockercfg-mw9tb\""
Oct 30 13:04:13.006337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.006315    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"]
Oct 30 13:04:13.079589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.079060    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a1ee32\""
Oct 30 13:04:13.164993 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.164365    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e689db56-2a1a-4de3-b73f-18f117c8cb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm\" (UID: \"e689db56-2a1a-4de3-b73f-18f117c8cb80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:13.164993 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.164477    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84j4\" (UniqueName: \"kubernetes.io/projected/e689db56-2a1a-4de3-b73f-18f117c8cb80-kube-api-access-m84j4\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm\" (UID: \"e689db56-2a1a-4de3-b73f-18f117c8cb80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:13.266337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.266227    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e689db56-2a1a-4de3-b73f-18f117c8cb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm\" (UID: \"e689db56-2a1a-4de3-b73f-18f117c8cb80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:13.266337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.266303    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84j4\" (UniqueName: \"kubernetes.io/projected/e689db56-2a1a-4de3-b73f-18f117c8cb80-kube-api-access-m84j4\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm\" (UID: \"e689db56-2a1a-4de3-b73f-18f117c8cb80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:13.266906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.266887    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e689db56-2a1a-4de3-b73f-18f117c8cb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm\" (UID: \"e689db56-2a1a-4de3-b73f-18f117c8cb80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:13.291773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.291749    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84j4\" (UniqueName: \"kubernetes.io/projected/e689db56-2a1a-4de3-b73f-18f117c8cb80-kube-api-access-m84j4\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm\" (UID: \"e689db56-2a1a-4de3-b73f-18f117c8cb80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:13.298111 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.298088    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:13.628864 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:04:13.628837    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode689db56_2a1a_4de3_b73f_18f117c8cb80.slice/crio-7759362a6ddd8d13188fb85df2d64680b70f371d7e736d1893992d4701717950 WatchSource:0}: Error finding container 7759362a6ddd8d13188fb85df2d64680b70f371d7e736d1893992d4701717950: Status 404 returned error can't find the container with id 7759362a6ddd8d13188fb85df2d64680b70f371d7e736d1893992d4701717950
Oct 30 13:04:13.631985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:13.631957    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"]
Oct 30 13:04:14.077252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:14.077223    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddffcdc2-3b27-4023-bd57-18e36871e8de" path="/var/lib/kubelet/pods/ddffcdc2-3b27-4023-bd57-18e36871e8de/volumes"
Oct 30 13:04:14.636916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:14.636885    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm" event={"ID":"e689db56-2a1a-4de3-b73f-18f117c8cb80","Type":"ContainerStarted","Data":"f4ff21407dbb10f8c6dcbb2d3d96a19ce58fab20d9ab5cb16912591d966fdd7b"}
Oct 30 13:04:14.637434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:14.636945    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm" event={"ID":"e689db56-2a1a-4de3-b73f-18f117c8cb80","Type":"ContainerStarted","Data":"7759362a6ddd8d13188fb85df2d64680b70f371d7e736d1893992d4701717950"}
Oct 30 13:04:14.637434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:14.637041    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:14.684962 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:14.684917    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm" podStartSLOduration=2.68490274 podStartE2EDuration="2.68490274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:14.682634683 +0000 UTC m=+1912.738128508" watchObservedRunningTime="2026-10-30 13:04:14.68490274 +0000 UTC m=+1912.740396563"
Oct 30 13:04:17.165709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.165662    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"]
Oct 30 13:04:17.170412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.170390    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.204327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.190328    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-405eb1-serving-cert\""
Oct 30 13:04:17.204327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.198800    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"]
Oct 30 13:04:17.219961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.219925    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.220124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.219971    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-metrics-certs\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.220124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.220100    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcj5h\" (UniqueName: \"kubernetes.io/projected/0116c41d-da7f-4cf9-9c19-91bf810ffc70-kube-api-access-lcj5h\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.220328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.220268    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0116c41d-da7f-4cf9-9c19-91bf810ffc70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.322062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.322026    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.322262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.322070    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-metrics-certs\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.322262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.322134    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcj5h\" (UniqueName: \"kubernetes.io/projected/0116c41d-da7f-4cf9-9c19-91bf810ffc70-kube-api-access-lcj5h\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.322262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.322236    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0116c41d-da7f-4cf9-9c19-91bf810ffc70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.322911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.322890    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0116c41d-da7f-4cf9-9c19-91bf810ffc70-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.324692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.324670    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-metrics-certs\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.324833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.324758    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-maas-api-tls\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.349833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.349799    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcj5h\" (UniqueName: \"kubernetes.io/projected/0116c41d-da7f-4cf9-9c19-91bf810ffc70-kube-api-access-lcj5h\") pod \"maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.488702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.488662    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:17.596707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.596676    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:17.814148 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:04:17.814115    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0116c41d_da7f_4cf9_9c19_91bf810ffc70.slice/crio-c9003d18b057a42c9da46da6b32593a8e69344406127ec07986065caaf4eced2 WatchSource:0}: Error finding container c9003d18b057a42c9da46da6b32593a8e69344406127ec07986065caaf4eced2: Status 404 returned error can't find the container with id c9003d18b057a42c9da46da6b32593a8e69344406127ec07986065caaf4eced2
Oct 30 13:04:17.814658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:17.814635    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"]
Oct 30 13:04:18.680668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:18.680560    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw" event={"ID":"0116c41d-da7f-4cf9-9c19-91bf810ffc70","Type":"ContainerStarted","Data":"a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375"}
Oct 30 13:04:18.680668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:18.680624    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw" event={"ID":"0116c41d-da7f-4cf9-9c19-91bf810ffc70","Type":"ContainerStarted","Data":"c9003d18b057a42c9da46da6b32593a8e69344406127ec07986065caaf4eced2"}
Oct 30 13:04:18.680668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:18.680650    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:18.733105 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:18.733045    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw" podStartSLOduration=1.234707986 podStartE2EDuration="1.733028221s" totalImagesPullingTime="498.320235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:18.727371284 +0000 UTC m=+1916.782865121" watchObservedRunningTime="2026-10-30 13:04:18.733028221 +0000 UTC m=+1916.788522045"
Oct 30 13:04:24.695606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:24.695576    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:04:25.226468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:25.226433    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"]
Oct 30 13:04:25.226667 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:25.226627    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy" containerID="cri-o://15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb" gracePeriod=30
Oct 30 13:04:25.646948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:25.646854    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:04:30.500500 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.500447    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 13:04:30.534392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534363    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-token\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-token\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534400    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-podinfo\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534507    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-credential-socket\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534554    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-data\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534589    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-certs\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534632    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-socket\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534663    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-envoy\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534700    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-kube-api-access-rckm5\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-kube-api-access-rckm5\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534813    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698f0851-42a2-4cf5-abfd-676cefceeef1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698f0851-42a2-4cf5-abfd-676cefceeef1-istiod-ca-cert\") pod \"698f0851-42a2-4cf5-abfd-676cefceeef1\" (UID: \"698f0851-42a2-4cf5-abfd-676cefceeef1\") "
Oct 30 13:04:30.534889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534841    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-data" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:30.534990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.534900    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-credential-socket" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:30.535040 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.535015    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-certs" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:30.535114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.535040    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-socket" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:30.535546 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.535520    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698f0851-42a2-4cf5-abfd-676cefceeef1-istiod-ca-cert" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:30.535706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.535603    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.535889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.535718    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.535889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.535737    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.535889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.535752    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.536786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.536766    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-token" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:30.536896 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.536880    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-podinfo" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:04:30.537161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.537145    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-envoy" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:30.537203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.537176    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-kube-api-access-rckm5" pod "698f0851-42a2-4cf5-abfd-676cefceeef1" (UID: "698f0851-42a2-4cf5-abfd-676cefceeef1"). InnerVolumeSpecName "kube-api-access-rckm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:30.636697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.636672    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.636697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.636695    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.636697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.636704    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698f0851-42a2-4cf5-abfd-676cefceeef1-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.636913 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.636712    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rckm5\" (UniqueName: \"kubernetes.io/projected/698f0851-42a2-4cf5-abfd-676cefceeef1-kube-api-access-rckm5\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.636913 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.636722    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698f0851-42a2-4cf5-abfd-676cefceeef1-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:30.775385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.775264    4429 generic.go:356] "Generic (PLEG): container finished" podID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerID="15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb" exitCode=0
Oct 30 13:04:30.775385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.775351    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"
Oct 30 13:04:30.775385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.775348    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" event={"ID":"698f0851-42a2-4cf5-abfd-676cefceeef1","Type":"ContainerDied","Data":"15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb"}
Oct 30 13:04:30.775635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.775413    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6" event={"ID":"698f0851-42a2-4cf5-abfd-676cefceeef1","Type":"ContainerDied","Data":"d5ffac9b5c4dbc4aa026d201c6eeaf15f7e70a25276a8a772736405318ede0c2"}
Oct 30 13:04:30.775635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.775438    4429 scope.go:122] "RemoveContainer" containerID="15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb"
Oct 30 13:04:30.787032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.787012    4429 scope.go:122] "RemoveContainer" containerID="15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb"
Oct 30 13:04:30.787314 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:30.787295    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb\": container with ID starting with 15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb not found: ID does not exist" containerID="15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb"
Oct 30 13:04:30.787365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.787323    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb"} err="failed to get container status \"15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb\": rpc error: code = NotFound desc = could not find container \"15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb\": container with ID starting with 15de3e0efad57ea62db2fc7540139fe65237acf796749bcd6569d2ed0847e4cb not found: ID does not exist"
Oct 30 13:04:30.847247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.847221    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"]
Oct 30 13:04:30.856896 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:30.856870    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-rf7z6"]
Oct 30 13:04:32.103921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:32.103663    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" path="/var/lib/kubelet/pods/698f0851-42a2-4cf5-abfd-676cefceeef1/volumes"
Oct 30 13:04:38.873501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:38.873465    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"]
Oct 30 13:04:38.874341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:38.874310    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt" podUID="fd6ced03-b5d9-4419-b70e-4d4dffa7183b" containerName="maas-api" containerID="cri-o://9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6" gracePeriod=30
Oct 30 13:04:39.326376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.326347    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:04:39.415343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.411883    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-openshift-service-ca\") pod \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") "
Oct 30 13:04:39.415343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.412005    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-kube-api-access-lrmnr\" (UniqueName: \"kubernetes.io/projected/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-kube-api-access-lrmnr\") pod \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") "
Oct 30 13:04:39.415343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.412055    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-metrics-certs\") pod \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") "
Oct 30 13:04:39.415343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.412186    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-maas-api-tls\") pod \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\" (UID: \"fd6ced03-b5d9-4419-b70e-4d4dffa7183b\") "
Oct 30 13:04:39.415343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.412326    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-openshift-service-ca" pod "fd6ced03-b5d9-4419-b70e-4d4dffa7183b" (UID: "fd6ced03-b5d9-4419-b70e-4d4dffa7183b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:39.415343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.412821    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:39.418499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.415725    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-maas-api-tls" pod "fd6ced03-b5d9-4419-b70e-4d4dffa7183b" (UID: "fd6ced03-b5d9-4419-b70e-4d4dffa7183b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:39.418499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.416635    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-kube-api-access-lrmnr" pod "fd6ced03-b5d9-4419-b70e-4d4dffa7183b" (UID: "fd6ced03-b5d9-4419-b70e-4d4dffa7183b"). InnerVolumeSpecName "kube-api-access-lrmnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:39.418499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.418380    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-metrics-certs" pod "fd6ced03-b5d9-4419-b70e-4d4dffa7183b" (UID: "fd6ced03-b5d9-4419-b70e-4d4dffa7183b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:39.516401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.514497    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrmnr\" (UniqueName: \"kubernetes.io/projected/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-kube-api-access-lrmnr\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:39.516401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.514532    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:39.516401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.514545    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd6ced03-b5d9-4419-b70e-4d4dffa7183b-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:39.859936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.859894    4429 generic.go:356] "Generic (PLEG): container finished" podID="fd6ced03-b5d9-4419-b70e-4d4dffa7183b" containerID="9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6" exitCode=0
Oct 30 13:04:39.860159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.859955    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"
Oct 30 13:04:39.860308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.860250    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt" event={"ID":"fd6ced03-b5d9-4419-b70e-4d4dffa7183b","Type":"ContainerDied","Data":"9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6"}
Oct 30 13:04:39.860489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.860338    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt" event={"ID":"fd6ced03-b5d9-4419-b70e-4d4dffa7183b","Type":"ContainerDied","Data":"ac5c00f7d8d2192b88107680330c3ae1f99fc0a365353486733645cf37bfba4c"}
Oct 30 13:04:39.860489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.860387    4429 scope.go:122] "RemoveContainer" containerID="9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6"
Oct 30 13:04:39.873031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.873010    4429 scope.go:122] "RemoveContainer" containerID="9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6"
Oct 30 13:04:39.873365 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:39.873340    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6\": container with ID starting with 9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6 not found: ID does not exist" containerID="9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6"
Oct 30 13:04:39.873483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.873372    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6"} err="failed to get container status \"9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6\": rpc error: code = NotFound desc = could not find container \"9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6\": container with ID starting with 9d39e8961f5252c672323629c6c09a2a3d6c3684bdcf70093cfd39c25182b7a6 not found: ID does not exist"
Oct 30 13:04:39.930707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.930615    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"]
Oct 30 13:04:39.942819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:39.942790    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2768f-579b589d47-phmwt"]
Oct 30 13:04:40.079020 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:40.078985    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6ced03-b5d9-4419-b70e-4d4dffa7183b" path="/var/lib/kubelet/pods/fd6ced03-b5d9-4419-b70e-4d4dffa7183b/volumes"
Oct 30 13:04:40.280900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:40.280868    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"]
Oct 30 13:04:40.281129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:40.281105    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy" containerID="cri-o://f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93" gracePeriod=30
Oct 30 13:04:41.924102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:41.924058    4429 generic.go:356] "Generic (PLEG): container finished" podID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerID="0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852" exitCode=0
Oct 30 13:04:41.924952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:41.924920    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" event={"ID":"51866e4a-2e91-47b3-bd71-99ca894f27d9","Type":"ContainerDied","Data":"0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852"}
Oct 30 13:04:41.925632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:41.925596    4429 scope.go:122] "RemoveContainer" containerID="0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852"
Oct 30 13:04:41.950964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:41.938512    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:41.950964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:41.938624    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:41.950964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:41.938675    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:42.374061 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:42.374023    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:04:42.933691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:42.933643    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" event={"ID":"51866e4a-2e91-47b3-bd71-99ca894f27d9","Type":"ContainerStarted","Data":"21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb"}
Oct 30 13:04:43.074139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:43.074101    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:04:43.074770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:43.074742    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:04:43.074908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:43.074796    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 13:04:43.681253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:43.680058    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"]
Oct 30 13:04:44.075313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:44.074484    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:04:44.075313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:44.074582    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 13:04:45.075465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.075419    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:04:45.075900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.075483    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 13:04:45.618456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.618432    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 13:04:45.787040 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.786994    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-kube-api-access-r9hfk\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-kube-api-access-r9hfk\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.787257 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.787118    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-podinfo\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.787257 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.787210    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istiod-ca-cert\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.787257 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.787253    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-certs\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.787466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.787317    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-credential-socket\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.787466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.787355    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-data\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.787466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.787408    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-token\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-token\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.787601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.787494    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-socket\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.787649 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.787618    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-envoy\") pod \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\" (UID: \"aaf11c9b-c168-4347-bd9b-c1c3aa757027\") "
Oct 30 13:04:45.788634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.788599    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-certs" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.789451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.789407    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.789657 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.789636    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-socket" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.789657 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.789650    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-credential-socket" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.790039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.789996    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-data" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.790245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.790222    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istiod-ca-cert" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:45.792558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.791932    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-kube-api-access-r9hfk" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "kube-api-access-r9hfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:45.794996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.794973    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-token" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:45.796072 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.796042    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-envoy" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:45.796615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.796578    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-podinfo" pod "aaf11c9b-c168-4347-bd9b-c1c3aa757027" (UID: "aaf11c9b-c168-4347-bd9b-c1c3aa757027"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:04:45.891102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.891070    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9hfk\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-kube-api-access-r9hfk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.891102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.891100    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.891102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.891110    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.891371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.891119    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.891371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.891128    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.891371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.891139    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.891371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.891151    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.891371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.891159    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf11c9b-c168-4347-bd9b-c1c3aa757027-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:45.980563 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.980525    4429 generic.go:356] "Generic (PLEG): container finished" podID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerID="f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93" exitCode=0
Oct 30 13:04:45.980760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.980576    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"
Oct 30 13:04:45.980760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.980604    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" event={"ID":"aaf11c9b-c168-4347-bd9b-c1c3aa757027","Type":"ContainerDied","Data":"f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93"}
Oct 30 13:04:45.980760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.980665    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd" event={"ID":"aaf11c9b-c168-4347-bd9b-c1c3aa757027","Type":"ContainerDied","Data":"30c83567f612eb79e71d4882b7255ccd9654df3947043b76a6840956254eba0a"}
Oct 30 13:04:45.980760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.980688    4429 scope.go:122] "RemoveContainer" containerID="f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93"
Oct 30 13:04:45.992607 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.992589    4429 scope.go:122] "RemoveContainer" containerID="f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93"
Oct 30 13:04:45.992929 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:45.992894    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93\": container with ID starting with f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93 not found: ID does not exist" containerID="f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93"
Oct 30 13:04:45.993027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:45.992938    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93"} err="failed to get container status \"f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93\": rpc error: code = NotFound desc = could not find container \"f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93\": container with ID starting with f045326380398208bf51f3e562ec7b7f8f617b1d669ff24456efb42084c11e93 not found: ID does not exist"
Oct 30 13:04:46.077360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.077015    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:04:46.077360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.077076    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 13:04:46.087666 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.087638    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"]
Oct 30 13:04:46.087814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.087678    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-cclzd"]
Oct 30 13:04:46.528890 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.528856    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:04:46.697306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.697255    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"]
Oct 30 13:04:46.698347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698319    4429 cpu_manager.go:414] "containerMap: removing container" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy"
Oct 30 13:04:46.698347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698346    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy"
Oct 30 13:04:46.698558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698379    4429 cpu_manager.go:414] "containerMap: removing container" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy"
Oct 30 13:04:46.698558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698387    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy"
Oct 30 13:04:46.698558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698437    4429 cpu_manager.go:414] "containerMap: removing container" podUID="fd6ced03-b5d9-4419-b70e-4d4dffa7183b" containerName="maas-api"
Oct 30 13:04:46.698558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698446    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6ced03-b5d9-4419-b70e-4d4dffa7183b" containerName="maas-api"
Oct 30 13:04:46.698757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698589    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" containerName="istio-proxy"
Oct 30 13:04:46.698757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698609    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="698f0851-42a2-4cf5-abfd-676cefceeef1" containerName="istio-proxy"
Oct 30 13:04:46.698757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.698625    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6ced03-b5d9-4419-b70e-4d4dffa7183b" containerName="maas-api"
Oct 30 13:04:46.704244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.704217    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.731624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.731594    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a121dd-openshift-default-dockercfg-t7vsr\""
Oct 30 13:04:46.736545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.736405    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"]
Oct 30 13:04:46.806191 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806095    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-data\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.806388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806268    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-token\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.806472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806440    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h926\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-kube-api-access-2h926\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.806529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806488    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-podinfo\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.806591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806568    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-credential-socket\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.806662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806612    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-envoy\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.806662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806653    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-certs\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.806743 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806705    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-socket\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.806804 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.806786    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istiod-ca-cert\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.934333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.910438    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-token\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.936879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.936760    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h926\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-kube-api-access-2h926\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.943503    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-podinfo\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.944070    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-credential-socket\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.944488    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-credential-socket\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.947448    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-envoy\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.947504    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-certs\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.947581    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-socket\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.947705    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istiod-ca-cert\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.947884    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-data\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.949306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.948469    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-data\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.982008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.962727    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istiod-ca-cert\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.982008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.963024    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-certs\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.982008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.972193    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-socket\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:46.982008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:46.981829    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-podinfo\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:47.028053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.010379    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h926\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-kube-api-access-2h926\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:47.028053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.010967    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-token\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:47.046140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.046101    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-envoy\") pod \"e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") " pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:47.074179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.074088    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:04:47.074489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.074458    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 13:04:47.319761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.319725    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:47.689442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.688813    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"]
Oct 30 13:04:47.692911 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:04:47.692887    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0161fb89_4a52_42c6_a804_e1fcf3d6c40e.slice/crio-5398bd6da7062e67e2170bbc4004493c09e29f657af53d6d219cf665323fe342 WatchSource:0}: Error finding container 5398bd6da7062e67e2170bbc4004493c09e29f657af53d6d219cf665323fe342: Status 404 returned error can't find the container with id 5398bd6da7062e67e2170bbc4004493c09e29f657af53d6d219cf665323fe342
Oct 30 13:04:47.696506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.696478    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:47.696643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.696628    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:47.696722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:47.696712    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:48.017381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.017231    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" event={"ID":"0161fb89-4a52-42c6-a804-e1fcf3d6c40e","Type":"ContainerStarted","Data":"39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2"}
Oct 30 13:04:48.017381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.017305    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" event={"ID":"0161fb89-4a52-42c6-a804-e1fcf3d6c40e","Type":"ContainerStarted","Data":"5398bd6da7062e67e2170bbc4004493c09e29f657af53d6d219cf665323fe342"}
Oct 30 13:04:48.067128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.067059    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" podStartSLOduration=2.067036784 podStartE2EDuration="2.067036784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:48.065750162 +0000 UTC m=+1946.121243987" watchObservedRunningTime="2026-10-30 13:04:48.067036784 +0000 UTC m=+1946.122530612"
Oct 30 13:04:48.077953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.075196    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 13:04:48.077953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.075265    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 13:04:48.082140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.082109    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf11c9b-c168-4347-bd9b-c1c3aa757027" path="/var/lib/kubelet/pods/aaf11c9b-c168-4347-bd9b-c1c3aa757027/volumes"
Oct 30 13:04:48.320785 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.320698    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:48.327039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.327012    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:48.686369 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:48.682372    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh" podUID="5948d55b-48cd-4a7f-a51e-c19a8c03740e" containerName="payload-pre-processing" containerID="cri-o://ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a" gracePeriod=25
Oct 30 13:04:49.011752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.011721    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:49.024869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.024835    4429 generic.go:356] "Generic (PLEG): container finished" podID="5948d55b-48cd-4a7f-a51e-c19a8c03740e" containerID="ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a" exitCode=0
Oct 30 13:04:49.025046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.024883    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"
Oct 30 13:04:49.025046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.024914    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh" event={"ID":"5948d55b-48cd-4a7f-a51e-c19a8c03740e","Type":"ContainerDied","Data":"ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a"}
Oct 30 13:04:49.025046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.024969    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh" event={"ID":"5948d55b-48cd-4a7f-a51e-c19a8c03740e","Type":"ContainerDied","Data":"5152ae285355a32a5b67a45a40f8fa1678dd704c7676e8bda21c17213fa8cd16"}
Oct 30 13:04:49.025046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.024995    4429 scope.go:122] "RemoveContainer" containerID="ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a"
Oct 30 13:04:49.025272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.025258    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:49.026408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.026382    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:04:49.035614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.035588    4429 scope.go:122] "RemoveContainer" containerID="ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a"
Oct 30 13:04:49.035893 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:04:49.035872    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a\": container with ID starting with ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a not found: ID does not exist" containerID="ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a"
Oct 30 13:04:49.035968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.035900    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a"} err="failed to get container status \"ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a\": rpc error: code = NotFound desc = could not find container \"ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a\": container with ID starting with ad054d2f875079297951c4a60f4e93454343b9936fb2b99ccdc32e620d86a92a not found: ID does not exist"
Oct 30 13:04:49.081767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.081737    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:04:49.084498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.084474    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:04:49.110861 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.110826    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5948d55b-48cd-4a7f-a51e-c19a8c03740e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5948d55b-48cd-4a7f-a51e-c19a8c03740e-plugins-config-volume\") pod \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\" (UID: \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\") "
Oct 30 13:04:49.111069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.111043    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5948d55b-48cd-4a7f-a51e-c19a8c03740e-kube-api-access-xthq8\" (UniqueName: \"kubernetes.io/projected/5948d55b-48cd-4a7f-a51e-c19a8c03740e-kube-api-access-xthq8\") pod \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\" (UID: \"5948d55b-48cd-4a7f-a51e-c19a8c03740e\") "
Oct 30 13:04:49.111345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.111267    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5948d55b-48cd-4a7f-a51e-c19a8c03740e-plugins-config-volume" pod "5948d55b-48cd-4a7f-a51e-c19a8c03740e" (UID: "5948d55b-48cd-4a7f-a51e-c19a8c03740e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:49.112018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.111993    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5948d55b-48cd-4a7f-a51e-c19a8c03740e-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:49.113482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.113455    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5948d55b-48cd-4a7f-a51e-c19a8c03740e-kube-api-access-xthq8" pod "5948d55b-48cd-4a7f-a51e-c19a8c03740e" (UID: "5948d55b-48cd-4a7f-a51e-c19a8c03740e"). InnerVolumeSpecName "kube-api-access-xthq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:49.214101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.214058    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xthq8\" (UniqueName: \"kubernetes.io/projected/5948d55b-48cd-4a7f-a51e-c19a8c03740e-kube-api-access-xthq8\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:04:49.414009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.413973    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"]
Oct 30 13:04:49.438901 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:49.438870    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-405eb1-86db5f6bc9-dgnkh"]
Oct 30 13:04:50.076681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:50.076648    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5948d55b-48cd-4a7f-a51e-c19a8c03740e" path="/var/lib/kubelet/pods/5948d55b-48cd-4a7f-a51e-c19a8c03740e/volumes"
Oct 30 13:04:52.719813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.719776    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"]
Oct 30 13:04:52.720487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.720470    4429 cpu_manager.go:414] "containerMap: removing container" podUID="5948d55b-48cd-4a7f-a51e-c19a8c03740e" containerName="payload-pre-processing"
Oct 30 13:04:52.720540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.720490    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5948d55b-48cd-4a7f-a51e-c19a8c03740e" containerName="payload-pre-processing"
Oct 30 13:04:52.720606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.720595    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="5948d55b-48cd-4a7f-a51e-c19a8c03740e" containerName="payload-pre-processing"
Oct 30 13:04:52.725587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.725566    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:52.754735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.754704    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"]
Oct 30 13:04:52.761696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.761661    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjj7\" (UniqueName: \"kubernetes.io/projected/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-kube-api-access-kdjj7\") pod \"payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm\" (UID: \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:52.761851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.761836    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm\" (UID: \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:52.762588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.762565    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:52.763032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.763012    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a121dd\""
Oct 30 13:04:52.763251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.763233    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a121dd-dockercfg-b97h4\""
Oct 30 13:04:52.769031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.769013    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"]
Oct 30 13:04:52.788155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.788014    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"]
Oct 30 13:04:52.863702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.863670    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345c221d-1715-4a01-aee5-92b5be2cf96d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a121dd-646987b794-9bfdk\" (UID: \"345c221d-1715-4a01-aee5-92b5be2cf96d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:52.863869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.863727    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm\" (UID: \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:52.863869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.863837    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz25p\" (UniqueName: \"kubernetes.io/projected/345c221d-1715-4a01-aee5-92b5be2cf96d-kube-api-access-bz25p\") pod \"payload-processing-e2e-api-b-a121dd-646987b794-9bfdk\" (UID: \"345c221d-1715-4a01-aee5-92b5be2cf96d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:52.864017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.863999    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjj7\" (UniqueName: \"kubernetes.io/projected/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-kube-api-access-kdjj7\") pod \"payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm\" (UID: \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:52.864399 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.864382    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm\" (UID: \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:52.891193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.891158    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjj7\" (UniqueName: \"kubernetes.io/projected/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-kube-api-access-kdjj7\") pod \"payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm\" (UID: \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:52.965925 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.965895    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345c221d-1715-4a01-aee5-92b5be2cf96d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a121dd-646987b794-9bfdk\" (UID: \"345c221d-1715-4a01-aee5-92b5be2cf96d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:52.966130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.965982    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz25p\" (UniqueName: \"kubernetes.io/projected/345c221d-1715-4a01-aee5-92b5be2cf96d-kube-api-access-bz25p\") pod \"payload-processing-e2e-api-b-a121dd-646987b794-9bfdk\" (UID: \"345c221d-1715-4a01-aee5-92b5be2cf96d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:52.966657 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.966632    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345c221d-1715-4a01-aee5-92b5be2cf96d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a121dd-646987b794-9bfdk\" (UID: \"345c221d-1715-4a01-aee5-92b5be2cf96d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:52.991868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:52.991804    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz25p\" (UniqueName: \"kubernetes.io/projected/345c221d-1715-4a01-aee5-92b5be2cf96d-kube-api-access-bz25p\") pod \"payload-processing-e2e-api-b-a121dd-646987b794-9bfdk\" (UID: \"345c221d-1715-4a01-aee5-92b5be2cf96d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:53.056309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:53.043073    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:53.102931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:53.082078    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:53.481339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:53.477617    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"]
Oct 30 13:04:53.486166 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:04:53.486131    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f91a6a6_9720_4e68_9cb0_e8424ed55f70.slice/crio-62aaa56e46cdc4f66539e32a7b4c7e4656ff5e5ad62bffa9aec1093334ee036e WatchSource:0}: Error finding container 62aaa56e46cdc4f66539e32a7b4c7e4656ff5e5ad62bffa9aec1093334ee036e: Status 404 returned error can't find the container with id 62aaa56e46cdc4f66539e32a7b4c7e4656ff5e5ad62bffa9aec1093334ee036e
Oct 30 13:04:53.513073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:53.513042    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"]
Oct 30 13:04:53.514319 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:04:53.514268    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345c221d_1715_4a01_aee5_92b5be2cf96d.slice/crio-889aca43f66fd7bb06f050daa1bfe4314fe5c6404afe0df1d44fb22c10080996 WatchSource:0}: Error finding container 889aca43f66fd7bb06f050daa1bfe4314fe5c6404afe0df1d44fb22c10080996: Status 404 returned error can't find the container with id 889aca43f66fd7bb06f050daa1bfe4314fe5c6404afe0df1d44fb22c10080996
Oct 30 13:04:54.218345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:54.208104    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm" event={"ID":"2f91a6a6-9720-4e68-9cb0-e8424ed55f70","Type":"ContainerStarted","Data":"ffc199e19d3b0c943fb817bd7d0736c551c301ca9aad0d2dac3b03f39e90d49d"}
Oct 30 13:04:54.218345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:54.217416    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm" event={"ID":"2f91a6a6-9720-4e68-9cb0-e8424ed55f70","Type":"ContainerStarted","Data":"62aaa56e46cdc4f66539e32a7b4c7e4656ff5e5ad62bffa9aec1093334ee036e"}
Oct 30 13:04:54.218923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:54.218642    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk" event={"ID":"345c221d-1715-4a01-aee5-92b5be2cf96d","Type":"ContainerStarted","Data":"d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0"}
Oct 30 13:04:54.218923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:54.218689    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk" event={"ID":"345c221d-1715-4a01-aee5-92b5be2cf96d","Type":"ContainerStarted","Data":"889aca43f66fd7bb06f050daa1bfe4314fe5c6404afe0df1d44fb22c10080996"}
Oct 30 13:04:54.220046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:54.219552    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:04:54.222309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:54.220817    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:04:54.271308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:54.270747    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk" podStartSLOduration=2.270728487 podStartE2EDuration="2.270728487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:54.268622516 +0000 UTC m=+1952.324116372" watchObservedRunningTime="2026-10-30 13:04:54.270728487 +0000 UTC m=+1952.326222312"
Oct 30 13:04:54.314876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:54.314825    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm" podStartSLOduration=2.314810152 podStartE2EDuration="2.314810152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:54.312473104 +0000 UTC m=+1952.367966951" watchObservedRunningTime="2026-10-30 13:04:54.314810152 +0000 UTC m=+1952.370303976"
Oct 30 13:04:57.116390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:04:57.116358    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"]
Oct 30 13:05:02.118847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:02.116913    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm" podUID="e689db56-2a1a-4de3-b73f-18f117c8cb80" containerName="payload-pre-processing" containerID="cri-o://f4ff21407dbb10f8c6dcbb2d3d96a19ce58fab20d9ab5cb16912591d966fdd7b" gracePeriod=25
Oct 30 13:05:02.335661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:02.335515    4429 generic.go:356] "Generic (PLEG): container finished" podID="e689db56-2a1a-4de3-b73f-18f117c8cb80" containerID="f4ff21407dbb10f8c6dcbb2d3d96a19ce58fab20d9ab5cb16912591d966fdd7b" exitCode=0
Oct 30 13:05:02.335661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:02.335592    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm" event={"ID":"e689db56-2a1a-4de3-b73f-18f117c8cb80","Type":"ContainerDied","Data":"f4ff21407dbb10f8c6dcbb2d3d96a19ce58fab20d9ab5cb16912591d966fdd7b"}
Oct 30 13:05:03.642311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:03.634762    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:05:03.804368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:03.804214    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e689db56-2a1a-4de3-b73f-18f117c8cb80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e689db56-2a1a-4de3-b73f-18f117c8cb80-plugins-config-volume\") pod \"e689db56-2a1a-4de3-b73f-18f117c8cb80\" (UID: \"e689db56-2a1a-4de3-b73f-18f117c8cb80\") "
Oct 30 13:05:03.812309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:03.804617    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e689db56-2a1a-4de3-b73f-18f117c8cb80-kube-api-access-m84j4\" (UniqueName: \"kubernetes.io/projected/e689db56-2a1a-4de3-b73f-18f117c8cb80-kube-api-access-m84j4\") pod \"e689db56-2a1a-4de3-b73f-18f117c8cb80\" (UID: \"e689db56-2a1a-4de3-b73f-18f117c8cb80\") "
Oct 30 13:05:03.837419 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:03.814137    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e689db56-2a1a-4de3-b73f-18f117c8cb80-plugins-config-volume" pod "e689db56-2a1a-4de3-b73f-18f117c8cb80" (UID: "e689db56-2a1a-4de3-b73f-18f117c8cb80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:03.845309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:03.843454    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e689db56-2a1a-4de3-b73f-18f117c8cb80-kube-api-access-m84j4" pod "e689db56-2a1a-4de3-b73f-18f117c8cb80" (UID: "e689db56-2a1a-4de3-b73f-18f117c8cb80"). InnerVolumeSpecName "kube-api-access-m84j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:03.910312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:03.909974    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84j4\" (UniqueName: \"kubernetes.io/projected/e689db56-2a1a-4de3-b73f-18f117c8cb80-kube-api-access-m84j4\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:03.910312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:03.910027    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e689db56-2a1a-4de3-b73f-18f117c8cb80-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:03.953310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:03.953092    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:04.052312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.051371    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:04.068361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.064579    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 13:05:04.068361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.065910    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:04.309312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.308370    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:04.403372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.402481    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:04.458311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.458059    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm" event={"ID":"e689db56-2a1a-4de3-b73f-18f117c8cb80","Type":"ContainerDied","Data":"7759362a6ddd8d13188fb85df2d64680b70f371d7e736d1893992d4701717950"}
Oct 30 13:05:04.458311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.458163    4429 scope.go:122] "RemoveContainer" containerID="f4ff21407dbb10f8c6dcbb2d3d96a19ce58fab20d9ab5cb16912591d966fdd7b"
Oct 30 13:05:04.458575 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.458534    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"
Oct 30 13:05:04.658362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.654410    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"]
Oct 30 13:05:04.680310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:04.659006    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-774dc7db59-xjbhm"]
Oct 30 13:05:05.067562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.067515    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:05.174312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.070885    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 13:05:05.174312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.158086    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:05:05.174312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.158149    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:05:05.174312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.077394    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:05.174312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.160605    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:05:05.385368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.385019    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:05:05.405311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.392530    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:05:05.405311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.402705    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 13:05:05.426027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.410476    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:05:05.495311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.481612    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:05:05.495311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:05.489183    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:05:06.104741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:06.104705    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e689db56-2a1a-4de3-b73f-18f117c8cb80" path="/var/lib/kubelet/pods/e689db56-2a1a-4de3-b73f-18f117c8cb80/volumes"
Oct 30 13:05:06.235220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:06.235194    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:05:06.235438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:06.235278    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:05:10.640431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:10.640388    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:10.640817 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:10.640442    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:11.528436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:11.528405    4429 generic.go:356] "Generic (PLEG): container finished" podID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerID="85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86" exitCode=0
Oct 30 13:05:11.528625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:11.528476    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" event={"ID":"657cfe5f-d9ae-42da-8649-e7de001439cd","Type":"ContainerDied","Data":"85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86"}
Oct 30 13:05:11.528928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:11.528912    4429 scope.go:122] "RemoveContainer" containerID="85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86"
Oct 30 13:05:11.530611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:11.530579    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:11.530726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:11.530652    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:11.530726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:11.530691    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:12.400329 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:05:12.400275    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:05:12.536777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:12.536743    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" event={"ID":"657cfe5f-d9ae-42da-8649-e7de001439cd","Type":"ContainerStarted","Data":"f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b"}
Oct 30 13:05:13.167309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:13.163626    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:05:13.167309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:13.163759    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:05:13.167309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:13.163817    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:05:14.170323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:14.169675    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:05:14.170323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:14.169746    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:05:15.152309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:15.151654    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:05:15.152309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:15.151716    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:05:16.172698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:16.172660    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:05:16.173356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:16.172739    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:05:17.158349 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:17.158311    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 13:05:17.158616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:17.158589    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 13:05:18.152715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:18.152683    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:05:18.154035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:18.154011    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:05:18.155327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:18.155275    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:05:19.058578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.058546    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"]
Oct 30 13:05:19.059859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.059815    4429 cpu_manager.go:414] "containerMap: removing container" podUID="e689db56-2a1a-4de3-b73f-18f117c8cb80" containerName="payload-pre-processing"
Oct 30 13:05:19.060054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.060036    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e689db56-2a1a-4de3-b73f-18f117c8cb80" containerName="payload-pre-processing"
Oct 30 13:05:19.060457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.060424    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="e689db56-2a1a-4de3-b73f-18f117c8cb80" containerName="payload-pre-processing"
Oct 30 13:05:19.075643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.070336    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"]
Oct 30 13:05:19.075643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.070443    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:19.104511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.104462    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a1ee32-dockercfg-8v2cm\""
Oct 30 13:05:19.105044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.105025    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a1ee32\""
Oct 30 13:05:19.154178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.154134    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"]
Oct 30 13:05:19.159022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.158991    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:19.160336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.160270    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2571c8-8d07-4569-a446-43c61d8632cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc\" (UID: \"db2571c8-8d07-4569-a446-43c61d8632cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:19.160508 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.160486    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693983ad-c737-45cd-9924-7285bf13b2a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn\" (UID: \"693983ad-c737-45cd-9924-7285bf13b2a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:19.160677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.160651    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wd5\" (UniqueName: \"kubernetes.io/projected/693983ad-c737-45cd-9924-7285bf13b2a6-kube-api-access-r8wd5\") pod \"payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn\" (UID: \"693983ad-c737-45cd-9924-7285bf13b2a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:19.160814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.160770    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29mz\" (UniqueName: \"kubernetes.io/projected/db2571c8-8d07-4569-a446-43c61d8632cb-kube-api-access-q29mz\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc\" (UID: \"db2571c8-8d07-4569-a446-43c61d8632cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:19.179032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.179008    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"]
Oct 30 13:05:19.261760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.261729    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693983ad-c737-45cd-9924-7285bf13b2a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn\" (UID: \"693983ad-c737-45cd-9924-7285bf13b2a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:19.261916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.261816    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wd5\" (UniqueName: \"kubernetes.io/projected/693983ad-c737-45cd-9924-7285bf13b2a6-kube-api-access-r8wd5\") pod \"payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn\" (UID: \"693983ad-c737-45cd-9924-7285bf13b2a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:19.261916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.261837    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29mz\" (UniqueName: \"kubernetes.io/projected/db2571c8-8d07-4569-a446-43c61d8632cb-kube-api-access-q29mz\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc\" (UID: \"db2571c8-8d07-4569-a446-43c61d8632cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:19.262019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.261922    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2571c8-8d07-4569-a446-43c61d8632cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc\" (UID: \"db2571c8-8d07-4569-a446-43c61d8632cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:19.262460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.262432    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693983ad-c737-45cd-9924-7285bf13b2a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn\" (UID: \"693983ad-c737-45cd-9924-7285bf13b2a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:19.262580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.262563    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2571c8-8d07-4569-a446-43c61d8632cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc\" (UID: \"db2571c8-8d07-4569-a446-43c61d8632cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:19.294583 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.294556    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wd5\" (UniqueName: \"kubernetes.io/projected/693983ad-c737-45cd-9924-7285bf13b2a6-kube-api-access-r8wd5\") pod \"payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn\" (UID: \"693983ad-c737-45cd-9924-7285bf13b2a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:19.294700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.294623    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29mz\" (UniqueName: \"kubernetes.io/projected/db2571c8-8d07-4569-a446-43c61d8632cb-kube-api-access-q29mz\") pod \"payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc\" (UID: \"db2571c8-8d07-4569-a446-43c61d8632cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:19.417823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.417748    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:19.480332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.473225    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:19.728331 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:05:19.728272    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2571c8_8d07_4569_a446_43c61d8632cb.slice/crio-f3a89a13632383bcaebf1b3152b2af6290cf589d70c739331d252b05b623fb83 WatchSource:0}: Error finding container f3a89a13632383bcaebf1b3152b2af6290cf589d70c739331d252b05b623fb83: Status 404 returned error can't find the container with id f3a89a13632383bcaebf1b3152b2af6290cf589d70c739331d252b05b623fb83
Oct 30 13:05:19.730093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.730071    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"]
Oct 30 13:05:19.869526 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:05:19.869487    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693983ad_c737_45cd_9924_7285bf13b2a6.slice/crio-9c636e698128ad0346f0ce918ef745ca131ac7061ed1c98344dbd42793745287 WatchSource:0}: Error finding container 9c636e698128ad0346f0ce918ef745ca131ac7061ed1c98344dbd42793745287: Status 404 returned error can't find the container with id 9c636e698128ad0346f0ce918ef745ca131ac7061ed1c98344dbd42793745287
Oct 30 13:05:19.870455 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:19.870428    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"]
Oct 30 13:05:20.625681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:20.625643    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn" event={"ID":"693983ad-c737-45cd-9924-7285bf13b2a6","Type":"ContainerStarted","Data":"fc87680d6783c806d0c7a6ceca2751965edcda83e0dc4b2b3645b6991fac2721"}
Oct 30 13:05:20.626201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:20.625720    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn" event={"ID":"693983ad-c737-45cd-9924-7285bf13b2a6","Type":"ContainerStarted","Data":"9c636e698128ad0346f0ce918ef745ca131ac7061ed1c98344dbd42793745287"}
Oct 30 13:05:20.626201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:20.625858    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:20.627702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:20.627676    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc" event={"ID":"db2571c8-8d07-4569-a446-43c61d8632cb","Type":"ContainerStarted","Data":"177eff8b0ff94d5d00afef8312c8f14827f8d61c051189d54875269debdaabcb"}
Oct 30 13:05:20.627847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:20.627717    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc" event={"ID":"db2571c8-8d07-4569-a446-43c61d8632cb","Type":"ContainerStarted","Data":"f3a89a13632383bcaebf1b3152b2af6290cf589d70c739331d252b05b623fb83"}
Oct 30 13:05:20.627847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:20.627749    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:20.673319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:20.673247    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn" podStartSLOduration=1.6732312390000001 podStartE2EDuration="1.673231239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:20.670059776 +0000 UTC m=+1978.725553601" watchObservedRunningTime="2026-10-30 13:05:20.673231239 +0000 UTC m=+1978.728725072"
Oct 30 13:05:20.718575 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:20.718509    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc" podStartSLOduration=2.718493757 podStartE2EDuration="2.718493757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:20.71553067 +0000 UTC m=+1978.771024498" watchObservedRunningTime="2026-10-30 13:05:20.718493757 +0000 UTC m=+1978.773987628"
Oct 30 13:05:23.823750 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:23.823715    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"]
Oct 30 13:05:23.824267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:23.823901    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy" containerID="cri-o://18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421" gracePeriod=30
Oct 30 13:05:23.837456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:23.837426    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"]
Oct 30 13:05:23.837656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:23.837635    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy" containerID="cri-o://94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f" gracePeriod=30
Oct 30 13:05:23.841083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:23.841054    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"]
Oct 30 13:05:23.841270 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:23.841247    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy" containerID="cri-o://ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e" gracePeriod=30
Oct 30 13:05:23.851178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:23.851151    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"]
Oct 30 13:05:23.851373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:23.851339    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy" containerID="cri-o://0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea" gracePeriod=30
Oct 30 13:05:29.151187 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.151160    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 13:05:29.213306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.200897    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.227529    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-data\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.227581    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-credential-socket\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.227629    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-kube-api-access-gg5zf\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-kube-api-access-gg5zf\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.227682    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-token\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-token\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.227851    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-certs\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.227900    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-podinfo\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.228057    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-envoy\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.228110    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-socket\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.228220    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3ca494-97b9-4fdc-8a51-672df9c78cda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3ca494-97b9-4fdc-8a51-672df9c78cda-istiod-ca-cert\") pod \"af3ca494-97b9-4fdc-8a51-672df9c78cda\" (UID: \"af3ca494-97b9-4fdc-8a51-672df9c78cda\") "
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.229275    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3ca494-97b9-4fdc-8a51-672df9c78cda-istiod-ca-cert" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.230138    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-socket" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.230182    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-credential-socket" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.230201    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-certs" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.230714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.230222    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-data" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.231585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.230939    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-kube-api-access-gg5zf" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "kube-api-access-gg5zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:29.234675 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.234649    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-podinfo" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:29.234783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.234712    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-token" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:29.236467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.236431    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-envoy" pod "af3ca494-97b9-4fdc-8a51-672df9c78cda" (UID: "af3ca494-97b9-4fdc-8a51-672df9c78cda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.244761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.244741    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 13:05:29.248874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.248857    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 13:05:29.329586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329555    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-kube-api-access-27wx5\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-kube-api-access-27wx5\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.329788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329620    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-data\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.329788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329649    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-token\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-token\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.329788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329687    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5335b07-5214-489c-a4b2-ba57cdfd1043-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5335b07-5214-489c-a4b2-ba57cdfd1043-istiod-ca-cert\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.329788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329728    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-envoy\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.329788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329767    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-kube-api-access-zhvzw\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-kube-api-access-zhvzw\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.330060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329848    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-socket\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.330060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329898    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-credential-socket\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.330060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329927    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-envoy\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.330060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329965    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aa2a529-057f-4803-964d-1cb779d28da0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2a529-057f-4803-964d-1cb779d28da0-istio-podinfo\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.330060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.329980    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-data" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.330060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330013    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed00f65-2134-46d3-9505-c12c7171eb33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed00f65-2134-46d3-9505-c12c7171eb33-istiod-ca-cert\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.330446 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330423    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-credential-socket" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.330557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330523    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5335b07-5214-489c-a4b2-ba57cdfd1043-istiod-ca-cert" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:29.330654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330633    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-socket" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.330706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330667    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-socket\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.330803 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330702    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-certs\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.330803 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330794    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-istio-token\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-istio-token\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.330911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330856    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-data\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.330911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330894    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aed00f65-2134-46d3-9505-c12c7171eb33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed00f65-2134-46d3-9505-c12c7171eb33-istio-podinfo\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.331008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330927    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-envoy\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.331008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.330965    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-socket\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.331113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331024    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-credential-socket\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.331113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331058    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-istio-token\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.331209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331173    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa2a529-057f-4803-964d-1cb779d28da0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2a529-057f-4803-964d-1cb779d28da0-istiod-ca-cert\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.331273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331207    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed00f65-2134-46d3-9505-c12c7171eb33-istiod-ca-cert" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:29.331273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331217    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-credential-socket\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.331273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331265    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-certs\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.331473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331346    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-certs\") pod \"aed00f65-2134-46d3-9505-c12c7171eb33\" (UID: \"aed00f65-2134-46d3-9505-c12c7171eb33\") "
Oct 30 13:05:29.331473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331448    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-data\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.331581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331482    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-kube-api-access-x57k4\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-kube-api-access-x57k4\") pod \"7aa2a529-057f-4803-964d-1cb779d28da0\" (UID: \"7aa2a529-057f-4803-964d-1cb779d28da0\") "
Oct 30 13:05:29.331581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.331561    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-podinfo\") pod \"c5335b07-5214-489c-a4b2-ba57cdfd1043\" (UID: \"c5335b07-5214-489c-a4b2-ba57cdfd1043\") "
Oct 30 13:05:29.332145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.332118    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-credential-socket" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.332395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.332370    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-socket" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.332744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.332721    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-credential-socket" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.333045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.333020    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa2a529-057f-4803-964d-1cb779d28da0-istiod-ca-cert" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:29.334445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.334423    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-data" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.334551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.334440    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-certs" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.334637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.334616    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-token" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:29.335504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.335475    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-certs" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.335606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.335495    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-data" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.335606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.335537    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-socket" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.335741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.335718    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-envoy" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.335820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.335793    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aa2a529-057f-4803-964d-1cb779d28da0-istio-podinfo" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:29.335960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.335907    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-envoy" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.336174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.336146    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-certs" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.336499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.336473    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-envoy" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:05:29.336942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.336917    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.336955    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.336970    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.336988    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337007    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337027    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337042    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3ca494-97b9-4fdc-8a51-672df9c78cda-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337057    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337071    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2a529-057f-4803-964d-1cb779d28da0-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337090    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337107    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337122    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337137    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5zf\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-kube-api-access-gg5zf\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337157    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337172    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337185    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337199    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337217    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337230    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5335b07-5214-489c-a4b2-ba57cdfd1043-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337244    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337259    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3ca494-97b9-4fdc-8a51-672df9c78cda-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337277    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337312    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3ca494-97b9-4fdc-8a51-672df9c78cda-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337326    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed00f65-2134-46d3-9505-c12c7171eb33-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.337335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337341    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.338323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337362    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa2a529-057f-4803-964d-1cb779d28da0-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.338323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337376    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed00f65-2134-46d3-9505-c12c7171eb33-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.338323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337391    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa2a529-057f-4803-964d-1cb779d28da0-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.338323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.337405    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5335b07-5214-489c-a4b2-ba57cdfd1043-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.340085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.340054    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-podinfo" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:29.342598 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.342572    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-kube-api-access-zhvzw" pod "c5335b07-5214-489c-a4b2-ba57cdfd1043" (UID: "c5335b07-5214-489c-a4b2-ba57cdfd1043"). InnerVolumeSpecName "kube-api-access-zhvzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:29.342920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.342897    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-kube-api-access-x57k4" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "kube-api-access-x57k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:29.351735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.351662    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-kube-api-access-27wx5" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "kube-api-access-27wx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:29.351735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.351745    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-istio-token" pod "7aa2a529-057f-4803-964d-1cb779d28da0" (UID: "7aa2a529-057f-4803-964d-1cb779d28da0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:29.351927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.351806    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-istio-token" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:29.354204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.354179    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aed00f65-2134-46d3-9505-c12c7171eb33-istio-podinfo" pod "aed00f65-2134-46d3-9505-c12c7171eb33" (UID: "aed00f65-2134-46d3-9505-c12c7171eb33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:05:29.438907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.438876    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvzw\" (UniqueName: \"kubernetes.io/projected/c5335b07-5214-489c-a4b2-ba57cdfd1043-kube-api-access-zhvzw\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.438907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.438907    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.439101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.438920    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed00f65-2134-46d3-9505-c12c7171eb33-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.439101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.438930    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.439101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.438941    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57k4\" (UniqueName: \"kubernetes.io/projected/7aa2a529-057f-4803-964d-1cb779d28da0-kube-api-access-x57k4\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.439101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.438953    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5335b07-5214-489c-a4b2-ba57cdfd1043-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.439101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.438964    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27wx5\" (UniqueName: \"kubernetes.io/projected/aed00f65-2134-46d3-9505-c12c7171eb33-kube-api-access-27wx5\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:29.701963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.701876    4429 generic.go:356] "Generic (PLEG): container finished" podID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerID="94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f" exitCode=0
Oct 30 13:05:29.701963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.701944    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"
Oct 30 13:05:29.701963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.701945    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" event={"ID":"c5335b07-5214-489c-a4b2-ba57cdfd1043","Type":"ContainerDied","Data":"94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f"}
Oct 30 13:05:29.702220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.702030    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn" event={"ID":"c5335b07-5214-489c-a4b2-ba57cdfd1043","Type":"ContainerDied","Data":"b1c320abd53f41bfcab6934e9b3301252383cc03b711fef8e7081b00025577d5"}
Oct 30 13:05:29.702220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.702060    4429 scope.go:122] "RemoveContainer" containerID="94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f"
Oct 30 13:05:29.708421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.708396    4429 generic.go:356] "Generic (PLEG): container finished" podID="7aa2a529-057f-4803-964d-1cb779d28da0" containerID="18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421" exitCode=0
Oct 30 13:05:29.708581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.708485    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"
Oct 30 13:05:29.708581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.708503    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" event={"ID":"7aa2a529-057f-4803-964d-1cb779d28da0","Type":"ContainerDied","Data":"18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421"}
Oct 30 13:05:29.708581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.708544    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp" event={"ID":"7aa2a529-057f-4803-964d-1cb779d28da0","Type":"ContainerDied","Data":"5bcf61c022c61636ae38265549d8bcc334d0e5522a20b0efb55ab842309dbe5c"}
Oct 30 13:05:29.710219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.710192    4429 generic.go:356] "Generic (PLEG): container finished" podID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerID="ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e" exitCode=0
Oct 30 13:05:29.710492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.710254    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"
Oct 30 13:05:29.710492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.710312    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" event={"ID":"af3ca494-97b9-4fdc-8a51-672df9c78cda","Type":"ContainerDied","Data":"ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e"}
Oct 30 13:05:29.710492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.710385    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s" event={"ID":"af3ca494-97b9-4fdc-8a51-672df9c78cda","Type":"ContainerDied","Data":"cd3ef01b0a4016729966a1de83e51a04b3f2ce5c4415e78810fe832ada1fe1ee"}
Oct 30 13:05:29.712148 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.712126    4429 generic.go:356] "Generic (PLEG): container finished" podID="aed00f65-2134-46d3-9505-c12c7171eb33" containerID="0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea" exitCode=0
Oct 30 13:05:29.712263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.712157    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" event={"ID":"aed00f65-2134-46d3-9505-c12c7171eb33","Type":"ContainerDied","Data":"0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea"}
Oct 30 13:05:29.712263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.712202    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"
Oct 30 13:05:29.712392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.712278    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" event={"ID":"aed00f65-2134-46d3-9505-c12c7171eb33","Type":"ContainerDied","Data":"e6eb0d56469ce93f17087fcd242c061968f56048e5af266a8b353b9a704c2f51"}
Oct 30 13:05:29.720786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.720759    4429 scope.go:122] "RemoveContainer" containerID="94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f"
Oct 30 13:05:29.721066 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:05:29.721045    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f\": container with ID starting with 94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f not found: ID does not exist" containerID="94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f"
Oct 30 13:05:29.721109 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.721075    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f"} err="failed to get container status \"94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f\": rpc error: code = NotFound desc = could not find container \"94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f\": container with ID starting with 94bd2fe49591a142d315e094e74e356d5f4c22b90feb4d54d4293a0dd440ef8f not found: ID does not exist"
Oct 30 13:05:29.721109 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.721102    4429 scope.go:122] "RemoveContainer" containerID="18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421"
Oct 30 13:05:29.731715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.731698    4429 scope.go:122] "RemoveContainer" containerID="18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421"
Oct 30 13:05:29.731970 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:05:29.731949    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421\": container with ID starting with 18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421 not found: ID does not exist" containerID="18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421"
Oct 30 13:05:29.732016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.731979    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421"} err="failed to get container status \"18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421\": rpc error: code = NotFound desc = could not find container \"18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421\": container with ID starting with 18c0cfde5a1b104c59aeb3f0d6d4f2ac77aab7daa75a8460e7a205ce0dae2421 not found: ID does not exist"
Oct 30 13:05:29.732016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.731994    4429 scope.go:122] "RemoveContainer" containerID="ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e"
Oct 30 13:05:29.740968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.740949    4429 scope.go:122] "RemoveContainer" containerID="ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e"
Oct 30 13:05:29.741242 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:05:29.741226    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e\": container with ID starting with ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e not found: ID does not exist" containerID="ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e"
Oct 30 13:05:29.741356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.741247    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e"} err="failed to get container status \"ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e\": rpc error: code = NotFound desc = could not find container \"ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e\": container with ID starting with ac75114a2d59268cdcbdd99792578969d837f8edb2d4a2e8bf4f23404df4714e not found: ID does not exist"
Oct 30 13:05:29.741356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.741264    4429 scope.go:122] "RemoveContainer" containerID="0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea"
Oct 30 13:05:29.750251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.750234    4429 scope.go:122] "RemoveContainer" containerID="0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea"
Oct 30 13:05:29.750534 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:05:29.750518    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea\": container with ID starting with 0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea not found: ID does not exist" containerID="0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea"
Oct 30 13:05:29.750579 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.750540    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea"} err="failed to get container status \"0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea\": rpc error: code = NotFound desc = could not find container \"0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea\": container with ID starting with 0e578e1c83807927ee413eeb9990a51fa15f151d2a95a4d202d2b9e9404717ea not found: ID does not exist"
Oct 30 13:05:29.814185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.814155    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"]
Oct 30 13:05:29.817338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.817313    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-flwwn"]
Oct 30 13:05:29.852870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.852845    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"]
Oct 30 13:05:29.855894 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.855753    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv"]
Oct 30 13:05:29.892417 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.892384    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"]
Oct 30 13:05:29.895350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.895330    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-9j9hp"]
Oct 30 13:05:29.934471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.934386    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"]
Oct 30 13:05:29.939911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:29.939883    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-dv87s"]
Oct 30 13:05:30.075538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:30.075461    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" path="/var/lib/kubelet/pods/7aa2a529-057f-4803-964d-1cb779d28da0/volumes"
Oct 30 13:05:30.075960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:30.075947    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" path="/var/lib/kubelet/pods/aed00f65-2134-46d3-9505-c12c7171eb33/volumes"
Oct 30 13:05:30.076387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:30.076374    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" path="/var/lib/kubelet/pods/af3ca494-97b9-4fdc-8a51-672df9c78cda/volumes"
Oct 30 13:05:30.077098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:30.077083    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" path="/var/lib/kubelet/pods/c5335b07-5214-489c-a4b2-ba57cdfd1043/volumes"
Oct 30 13:05:30.131825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:30.131790    4429 patch_prober.go:29] interesting pod/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:30.131975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:30.131843    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e652ca-openshift-default-7b8fb7f765-26sgv" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:31.638963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:31.638924    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:05:31.639493 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:31.639423    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:05:39.222948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.222913    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"]
Oct 30 13:05:39.223590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.223147    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt" podUID="670fae55-f8db-47ed-8b56-e9974b2215d8" containerName="payload-processing" containerID="cri-o://0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea" gracePeriod=30
Oct 30 13:05:39.263605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.263573    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"]
Oct 30 13:05:39.263908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.263859    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww" podUID="522c084f-7798-48c9-91b6-b0c26e39c49b" containerName="payload-pre-processing" containerID="cri-o://b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496" gracePeriod=30
Oct 30 13:05:39.739827 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.739790    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:05:39.750942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.750912    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:05:39.820676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.820588    4429 generic.go:356] "Generic (PLEG): container finished" podID="522c084f-7798-48c9-91b6-b0c26e39c49b" containerID="b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496" exitCode=0
Oct 30 13:05:39.820676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.820636    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww" event={"ID":"522c084f-7798-48c9-91b6-b0c26e39c49b","Type":"ContainerDied","Data":"b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496"}
Oct 30 13:05:39.820676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.820649    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"
Oct 30 13:05:39.820958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.820692    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww" event={"ID":"522c084f-7798-48c9-91b6-b0c26e39c49b","Type":"ContainerDied","Data":"2c9aed291e52292b984fb824ce0310c08c45bc0a797fc7fbf300d4fcc4cd5d2d"}
Oct 30 13:05:39.820958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.820722    4429 scope.go:122] "RemoveContainer" containerID="b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496"
Oct 30 13:05:39.822692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.822660    4429 generic.go:356] "Generic (PLEG): container finished" podID="670fae55-f8db-47ed-8b56-e9974b2215d8" containerID="0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea" exitCode=0
Oct 30 13:05:39.822847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.822828    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt" event={"ID":"670fae55-f8db-47ed-8b56-e9974b2215d8","Type":"ContainerDied","Data":"0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea"}
Oct 30 13:05:39.822932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.822879    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt" event={"ID":"670fae55-f8db-47ed-8b56-e9974b2215d8","Type":"ContainerDied","Data":"bded35a5b5178057e15bcdc49a6a66e807288a26103d4c6a8e388865dbcb9cb6"}
Oct 30 13:05:39.822989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.822959    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"
Oct 30 13:05:39.836982 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.836955    4429 scope.go:122] "RemoveContainer" containerID="b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496"
Oct 30 13:05:39.837439 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:05:39.837417    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496\": container with ID starting with b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496 not found: ID does not exist" containerID="b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496"
Oct 30 13:05:39.837538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.837452    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496"} err="failed to get container status \"b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496\": rpc error: code = NotFound desc = could not find container \"b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496\": container with ID starting with b5381d06a90d78662b355513706b0d13d69214fbb60e2d1ca2924e7dc415d496 not found: ID does not exist"
Oct 30 13:05:39.837765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.837746    4429 scope.go:122] "RemoveContainer" containerID="0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea"
Oct 30 13:05:39.849319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.849208    4429 scope.go:122] "RemoveContainer" containerID="0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea"
Oct 30 13:05:39.849703 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:05:39.849676    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea\": container with ID starting with 0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea not found: ID does not exist" containerID="0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea"
Oct 30 13:05:39.849787 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.849712    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea"} err="failed to get container status \"0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea\": rpc error: code = NotFound desc = could not find container \"0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea\": container with ID starting with 0a027060a959ec19a8f58c2d759bcac5e771a991e0894a19d9328c59ae3153ea not found: ID does not exist"
Oct 30 13:05:39.852158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.852140    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522c084f-7798-48c9-91b6-b0c26e39c49b-kube-api-access-fsvp2\" (UniqueName: \"kubernetes.io/projected/522c084f-7798-48c9-91b6-b0c26e39c49b-kube-api-access-fsvp2\") pod \"522c084f-7798-48c9-91b6-b0c26e39c49b\" (UID: \"522c084f-7798-48c9-91b6-b0c26e39c49b\") "
Oct 30 13:05:39.852249 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.852219    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522c084f-7798-48c9-91b6-b0c26e39c49b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c084f-7798-48c9-91b6-b0c26e39c49b-plugins-config-volume\") pod \"522c084f-7798-48c9-91b6-b0c26e39c49b\" (UID: \"522c084f-7798-48c9-91b6-b0c26e39c49b\") "
Oct 30 13:05:39.852371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.852353    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670fae55-f8db-47ed-8b56-e9974b2215d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fae55-f8db-47ed-8b56-e9974b2215d8-plugins-config-volume\") pod \"670fae55-f8db-47ed-8b56-e9974b2215d8\" (UID: \"670fae55-f8db-47ed-8b56-e9974b2215d8\") "
Oct 30 13:05:39.852437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.852406    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670fae55-f8db-47ed-8b56-e9974b2215d8-kube-api-access-bzx6x\" (UniqueName: \"kubernetes.io/projected/670fae55-f8db-47ed-8b56-e9974b2215d8-kube-api-access-bzx6x\") pod \"670fae55-f8db-47ed-8b56-e9974b2215d8\" (UID: \"670fae55-f8db-47ed-8b56-e9974b2215d8\") "
Oct 30 13:05:39.852626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.852599    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522c084f-7798-48c9-91b6-b0c26e39c49b-plugins-config-volume" pod "522c084f-7798-48c9-91b6-b0c26e39c49b" (UID: "522c084f-7798-48c9-91b6-b0c26e39c49b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:39.852891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.852860    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670fae55-f8db-47ed-8b56-e9974b2215d8-plugins-config-volume" pod "670fae55-f8db-47ed-8b56-e9974b2215d8" (UID: "670fae55-f8db-47ed-8b56-e9974b2215d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:05:39.853120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.853098    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c084f-7798-48c9-91b6-b0c26e39c49b-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:39.853193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.853127    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fae55-f8db-47ed-8b56-e9974b2215d8-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:39.855107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.855080    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670fae55-f8db-47ed-8b56-e9974b2215d8-kube-api-access-bzx6x" pod "670fae55-f8db-47ed-8b56-e9974b2215d8" (UID: "670fae55-f8db-47ed-8b56-e9974b2215d8"). InnerVolumeSpecName "kube-api-access-bzx6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:39.855189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.855132    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522c084f-7798-48c9-91b6-b0c26e39c49b-kube-api-access-fsvp2" pod "522c084f-7798-48c9-91b6-b0c26e39c49b" (UID: "522c084f-7798-48c9-91b6-b0c26e39c49b"). InnerVolumeSpecName "kube-api-access-fsvp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:05:39.954340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.954279    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzx6x\" (UniqueName: \"kubernetes.io/projected/670fae55-f8db-47ed-8b56-e9974b2215d8-kube-api-access-bzx6x\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:39.954340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:39.954339    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsvp2\" (UniqueName: \"kubernetes.io/projected/522c084f-7798-48c9-91b6-b0c26e39c49b-kube-api-access-fsvp2\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:05:40.168621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.168590    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"]
Oct 30 13:05:40.182421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.182394    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9368ce-ddbf89db7-zsxww"]
Oct 30 13:05:40.210669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.210628    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"]
Oct 30 13:05:40.214124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.214100    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9368ce-6bc4c59674-vhsbt"]
Oct 30 13:05:40.833866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.833829    4429 generic.go:356] "Generic (PLEG): container finished" podID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerID="39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2" exitCode=0
Oct 30 13:05:40.834468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.833894    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" event={"ID":"0161fb89-4a52-42c6-a804-e1fcf3d6c40e","Type":"ContainerDied","Data":"39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2"}
Oct 30 13:05:40.834468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.834417    4429 scope.go:122] "RemoveContainer" containerID="39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2"
Oct 30 13:05:40.837490 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.837449    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:40.837677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.837621    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:40.837750 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:40.837733    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:41.883699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:41.883655    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" event={"ID":"0161fb89-4a52-42c6-a804-e1fcf3d6c40e","Type":"ContainerStarted","Data":"824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510"}
Oct 30 13:05:42.080807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:42.080765    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522c084f-7798-48c9-91b6-b0c26e39c49b" path="/var/lib/kubelet/pods/522c084f-7798-48c9-91b6-b0c26e39c49b/volumes"
Oct 30 13:05:42.081240 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:42.081219    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670fae55-f8db-47ed-8b56-e9974b2215d8" path="/var/lib/kubelet/pods/670fae55-f8db-47ed-8b56-e9974b2215d8/volumes"
Oct 30 13:05:42.320174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:42.320145    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:05:42.320425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:42.320390    4429 patch_prober.go:29] interesting pod/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:05:42.320495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:42.320457    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:05:42.399573 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:05:42.399529    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:05:43.321355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:43.321318    4429 patch_prober.go:29] interesting pod/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:05:43.321805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:43.321387    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:05:44.320655 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:44.320622    4429 patch_prober.go:29] interesting pod/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:05:44.320832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:44.320672    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:05:45.321172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:45.321137    4429 patch_prober.go:29] interesting pod/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:05:45.321605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:45.321194    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:05:46.320457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:46.320423    4429 patch_prober.go:29] interesting pod/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 13:05:46.320635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:46.320476    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 13:05:47.325139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:47.325108    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:05:47.325592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:47.325451    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:05:47.326278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:05:47.326256    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:06:00.361720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.361677    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"]
Oct 30 13:06:00.362179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.361892    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl" podUID="7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8" containerName="payload-processing" containerID="cri-o://6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1" gracePeriod=30
Oct 30 13:06:00.380960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.380929    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"]
Oct 30 13:06:00.381323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.381263    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n" podUID="6f0c4a8b-81f8-40c4-b23a-8bef204310da" containerName="payload-pre-processing" containerID="cri-o://498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650" gracePeriod=30
Oct 30 13:06:00.667607 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.667570    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:06:00.769849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.769809    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-kube-api-access-kvdr2\" (UniqueName: \"kubernetes.io/projected/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-kube-api-access-kvdr2\") pod \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\" (UID: \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\") "
Oct 30 13:06:00.770060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.770004    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-plugins-config-volume\") pod \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\" (UID: \"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8\") "
Oct 30 13:06:00.770648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.770600    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-plugins-config-volume" pod "7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8" (UID: "7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:00.793817 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.793749    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-kube-api-access-kvdr2" pod "7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8" (UID: "7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8"). InnerVolumeSpecName "kube-api-access-kvdr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:00.820766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.820730    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:06:00.871936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.871854    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0c4a8b-81f8-40c4-b23a-8bef204310da-kube-api-access-w2rgs\" (UniqueName: \"kubernetes.io/projected/6f0c4a8b-81f8-40c4-b23a-8bef204310da-kube-api-access-w2rgs\") pod \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\" (UID: \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\") "
Oct 30 13:06:00.871936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.871917    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0c4a8b-81f8-40c4-b23a-8bef204310da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0c4a8b-81f8-40c4-b23a-8bef204310da-plugins-config-volume\") pod \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\" (UID: \"6f0c4a8b-81f8-40c4-b23a-8bef204310da\") "
Oct 30 13:06:00.872331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.872306    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0c4a8b-81f8-40c4-b23a-8bef204310da-plugins-config-volume" pod "6f0c4a8b-81f8-40c4-b23a-8bef204310da" (UID: "6f0c4a8b-81f8-40c4-b23a-8bef204310da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:00.872484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.872468    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:00.872548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.872490    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0c4a8b-81f8-40c4-b23a-8bef204310da-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:00.872548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.872505    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvdr2\" (UniqueName: \"kubernetes.io/projected/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8-kube-api-access-kvdr2\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:00.874074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.874055    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0c4a8b-81f8-40c4-b23a-8bef204310da-kube-api-access-w2rgs" pod "6f0c4a8b-81f8-40c4-b23a-8bef204310da" (UID: "6f0c4a8b-81f8-40c4-b23a-8bef204310da"). InnerVolumeSpecName "kube-api-access-w2rgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:00.974214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:00.974176    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rgs\" (UniqueName: \"kubernetes.io/projected/6f0c4a8b-81f8-40c4-b23a-8bef204310da-kube-api-access-w2rgs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:01.043847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.043814    4429 generic.go:356] "Generic (PLEG): container finished" podID="7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8" containerID="6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1" exitCode=0
Oct 30 13:06:01.044006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.043866    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"
Oct 30 13:06:01.044006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.043901    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl" event={"ID":"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8","Type":"ContainerDied","Data":"6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1"}
Oct 30 13:06:01.044006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.043960    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl" event={"ID":"7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8","Type":"ContainerDied","Data":"6d9294d2868b9410736f5d535523dc97e3437523f6242419d3d91703964b3abb"}
Oct 30 13:06:01.044006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.043987    4429 scope.go:122] "RemoveContainer" containerID="6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1"
Oct 30 13:06:01.045742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.045720    4429 generic.go:356] "Generic (PLEG): container finished" podID="6f0c4a8b-81f8-40c4-b23a-8bef204310da" containerID="498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650" exitCode=0
Oct 30 13:06:01.045874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.045770    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"
Oct 30 13:06:01.045874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.045825    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n" event={"ID":"6f0c4a8b-81f8-40c4-b23a-8bef204310da","Type":"ContainerDied","Data":"498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650"}
Oct 30 13:06:01.045874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.045865    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n" event={"ID":"6f0c4a8b-81f8-40c4-b23a-8bef204310da","Type":"ContainerDied","Data":"b819d6a8384a45ad8e98e52c26a1579e85bd25aebfb5bc01536439319c692402"}
Oct 30 13:06:01.058161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.055579    4429 scope.go:122] "RemoveContainer" containerID="6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1"
Oct 30 13:06:01.065725 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:01.065687    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1\": container with ID starting with 6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1 not found: ID does not exist" containerID="6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1"
Oct 30 13:06:01.065847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.065738    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1"} err="failed to get container status \"6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1\": rpc error: code = NotFound desc = could not find container \"6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1\": container with ID starting with 6a7bbf511aa1446a72f07988a3e7d3d898a78b5a772a3b90841018a607657fc1 not found: ID does not exist"
Oct 30 13:06:01.065847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.065765    4429 scope.go:122] "RemoveContainer" containerID="498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650"
Oct 30 13:06:01.075512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.075497    4429 scope.go:122] "RemoveContainer" containerID="498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650"
Oct 30 13:06:01.075781 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:01.075762    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650\": container with ID starting with 498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650 not found: ID does not exist" containerID="498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650"
Oct 30 13:06:01.075833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.075788    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650"} err="failed to get container status \"498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650\": rpc error: code = NotFound desc = could not find container \"498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650\": container with ID starting with 498d1649642c52e1a34ef827183fa5348f40f1c24c5686e10d021fe08c200650 not found: ID does not exist"
Oct 30 13:06:01.128392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.128312    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"]
Oct 30 13:06:01.132660 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.132639    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f0f50-698cd75fc5-rmt4n"]
Oct 30 13:06:01.172695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.172658    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"]
Oct 30 13:06:01.195934 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:01.194479    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f0f50-668d688948-srbpl"]
Oct 30 13:06:02.079211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:02.079179    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0c4a8b-81f8-40c4-b23a-8bef204310da" path="/var/lib/kubelet/pods/6f0c4a8b-81f8-40c4-b23a-8bef204310da/volumes"
Oct 30 13:06:02.079700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:02.079681    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8" path="/var/lib/kubelet/pods/7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8/volumes"
Oct 30 13:06:10.449548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.449469    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"]
Oct 30 13:06:10.450103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450087    4429 cpu_manager.go:414] "containerMap: removing container" podUID="6f0c4a8b-81f8-40c4-b23a-8bef204310da" containerName="payload-pre-processing"
Oct 30 13:06:10.450161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450103    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0c4a8b-81f8-40c4-b23a-8bef204310da" containerName="payload-pre-processing"
Oct 30 13:06:10.450161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450120    4429 cpu_manager.go:414] "containerMap: removing container" podUID="522c084f-7798-48c9-91b6-b0c26e39c49b" containerName="payload-pre-processing"
Oct 30 13:06:10.450161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450125    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522c084f-7798-48c9-91b6-b0c26e39c49b" containerName="payload-pre-processing"
Oct 30 13:06:10.450161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450134    4429 cpu_manager.go:414] "containerMap: removing container" podUID="7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8" containerName="payload-processing"
Oct 30 13:06:10.450161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450139    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8" containerName="payload-processing"
Oct 30 13:06:10.450161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450149    4429 cpu_manager.go:414] "containerMap: removing container" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy"
Oct 30 13:06:10.450161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450153    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450175    4429 cpu_manager.go:414] "containerMap: removing container" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450180    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450191    4429 cpu_manager.go:414] "containerMap: removing container" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450195    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450202    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450206    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450224    4429 cpu_manager.go:414] "containerMap: removing container" podUID="670fae55-f8db-47ed-8b56-e9974b2215d8" containerName="payload-processing"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450228    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670fae55-f8db-47ed-8b56-e9974b2215d8" containerName="payload-processing"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450333    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa2a529-057f-4803-964d-1cb779d28da0" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450344    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5335b07-5214-489c-a4b2-ba57cdfd1043" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450352    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="670fae55-f8db-47ed-8b56-e9974b2215d8" containerName="payload-processing"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450361    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0c4a8b-81f8-40c4-b23a-8bef204310da" containerName="payload-pre-processing"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450366    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="522c084f-7798-48c9-91b6-b0c26e39c49b" containerName="payload-pre-processing"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450373    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c41a54c-071c-4c7c-ac6d-0cee9dc36ee8" containerName="payload-processing"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450378    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed00f65-2134-46d3-9505-c12c7171eb33" containerName="istio-proxy"
Oct 30 13:06:10.450387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.450385    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3ca494-97b9-4fdc-8a51-672df9c78cda" containerName="istio-proxy"
Oct 30 13:06:10.455573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.455551    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:10.473884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.473856    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"]
Oct 30 13:06:10.484755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.484725    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f9c553-dockercfg-b7rfb\""
Oct 30 13:06:10.485466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.485444    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f9c553\""
Oct 30 13:06:10.607780 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.607745    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b7e09-67f1-4df1-b385-ae9ae8370634-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2\" (UID: \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:10.607977 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.607952    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnll5\" (UniqueName: \"kubernetes.io/projected/2a1b7e09-67f1-4df1-b385-ae9ae8370634-kube-api-access-bnll5\") pod \"payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2\" (UID: \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:10.709805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.709724    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b7e09-67f1-4df1-b385-ae9ae8370634-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2\" (UID: \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:10.709955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.709853    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnll5\" (UniqueName: \"kubernetes.io/projected/2a1b7e09-67f1-4df1-b385-ae9ae8370634-kube-api-access-bnll5\") pod \"payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2\" (UID: \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:10.710502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.710482    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b7e09-67f1-4df1-b385-ae9ae8370634-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2\" (UID: \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:10.735943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.735916    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnll5\" (UniqueName: \"kubernetes.io/projected/2a1b7e09-67f1-4df1-b385-ae9ae8370634-kube-api-access-bnll5\") pod \"payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2\" (UID: \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:10.768899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:10.768869    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:11.100543 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:06:11.100510    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1b7e09_67f1_4df1_b385_ae9ae8370634.slice/crio-ebce32220279c30fb9a69763bb6971a6bae5cff8d71d33167f07e539deee91eb WatchSource:0}: Error finding container ebce32220279c30fb9a69763bb6971a6bae5cff8d71d33167f07e539deee91eb: Status 404 returned error can't find the container with id ebce32220279c30fb9a69763bb6971a6bae5cff8d71d33167f07e539deee91eb
Oct 30 13:06:11.101574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:11.101547    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"]
Oct 30 13:06:11.121763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:11.121732    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2" event={"ID":"2a1b7e09-67f1-4df1-b385-ae9ae8370634","Type":"ContainerStarted","Data":"ebce32220279c30fb9a69763bb6971a6bae5cff8d71d33167f07e539deee91eb"}
Oct 30 13:06:12.130640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:12.130595    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2" event={"ID":"2a1b7e09-67f1-4df1-b385-ae9ae8370634","Type":"ContainerStarted","Data":"9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4"}
Oct 30 13:06:12.131074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:12.130780    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:12.177443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:12.177382    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2" podStartSLOduration=2.177367793 podStartE2EDuration="2.177367793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:12.175299661 +0000 UTC m=+2030.230793479" watchObservedRunningTime="2026-10-30 13:06:12.177367793 +0000 UTC m=+2030.232861633"
Oct 30 13:06:12.375573 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:12.375530    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:06:15.622610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:15.622581    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"]
Oct 30 13:06:15.623038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:15.622804    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw" podUID="0116c41d-da7f-4cf9-9c19-91bf810ffc70" containerName="maas-api" containerID="cri-o://a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375" gracePeriod=30
Oct 30 13:06:15.923922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:15.923894    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:06:16.093317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.093273    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-metrics-certs\") pod \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") "
Oct 30 13:06:16.093553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.093536    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0116c41d-da7f-4cf9-9c19-91bf810ffc70-kube-api-access-lcj5h\" (UniqueName: \"kubernetes.io/projected/0116c41d-da7f-4cf9-9c19-91bf810ffc70-kube-api-access-lcj5h\") pod \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") "
Oct 30 13:06:16.093673 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.093654    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-maas-api-tls\") pod \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") "
Oct 30 13:06:16.093829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.093692    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0116c41d-da7f-4cf9-9c19-91bf810ffc70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0116c41d-da7f-4cf9-9c19-91bf810ffc70-openshift-service-ca\") pod \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\" (UID: \"0116c41d-da7f-4cf9-9c19-91bf810ffc70\") "
Oct 30 13:06:16.094232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.094205    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0116c41d-da7f-4cf9-9c19-91bf810ffc70-openshift-service-ca" pod "0116c41d-da7f-4cf9-9c19-91bf810ffc70" (UID: "0116c41d-da7f-4cf9-9c19-91bf810ffc70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:16.094497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.094478    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0116c41d-da7f-4cf9-9c19-91bf810ffc70-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:16.096739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.096236    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-metrics-certs" pod "0116c41d-da7f-4cf9-9c19-91bf810ffc70" (UID: "0116c41d-da7f-4cf9-9c19-91bf810ffc70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:16.096739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.096686    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0116c41d-da7f-4cf9-9c19-91bf810ffc70-kube-api-access-lcj5h" pod "0116c41d-da7f-4cf9-9c19-91bf810ffc70" (UID: "0116c41d-da7f-4cf9-9c19-91bf810ffc70"). InnerVolumeSpecName "kube-api-access-lcj5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:16.097331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.097272    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-maas-api-tls" pod "0116c41d-da7f-4cf9-9c19-91bf810ffc70" (UID: "0116c41d-da7f-4cf9-9c19-91bf810ffc70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:06:16.213793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.213757    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:16.215549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.215507    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcj5h\" (UniqueName: \"kubernetes.io/projected/0116c41d-da7f-4cf9-9c19-91bf810ffc70-kube-api-access-lcj5h\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:16.216789 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.216763    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0116c41d-da7f-4cf9-9c19-91bf810ffc70-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:16.216920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.216606    4429 generic.go:356] "Generic (PLEG): container finished" podID="0116c41d-da7f-4cf9-9c19-91bf810ffc70" containerID="a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375" exitCode=0
Oct 30 13:06:16.216980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.216701    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"
Oct 30 13:06:16.221231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.216635    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw" event={"ID":"0116c41d-da7f-4cf9-9c19-91bf810ffc70","Type":"ContainerDied","Data":"a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375"}
Oct 30 13:06:16.221231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.221089    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw" event={"ID":"0116c41d-da7f-4cf9-9c19-91bf810ffc70","Type":"ContainerDied","Data":"c9003d18b057a42c9da46da6b32593a8e69344406127ec07986065caaf4eced2"}
Oct 30 13:06:16.221231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.221123    4429 scope.go:122] "RemoveContainer" containerID="a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375"
Oct 30 13:06:16.251243 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.251219    4429 scope.go:122] "RemoveContainer" containerID="a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375"
Oct 30 13:06:16.251596 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:16.251577    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375\": container with ID starting with a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375 not found: ID does not exist" containerID="a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375"
Oct 30 13:06:16.251661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.251605    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375"} err="failed to get container status \"a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375\": rpc error: code = NotFound desc = could not find container \"a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375\": container with ID starting with a97da7737ed8ddf061a07c198f9ee8364e4cd61c91c3b08a2c2db073fe127375 not found: ID does not exist"
Oct 30 13:06:16.288553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.288528    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"]
Oct 30 13:06:16.292187 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:16.292166    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-405eb1-6c6df56df-j4kjw"]
Oct 30 13:06:18.075532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:18.075504    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0116c41d-da7f-4cf9-9c19-91bf810ffc70" path="/var/lib/kubelet/pods/0116c41d-da7f-4cf9-9c19-91bf810ffc70/volumes"
Oct 30 13:06:21.864749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:21.864709    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"]
Oct 30 13:06:21.865251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:21.864933    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn" podUID="693983ad-c737-45cd-9924-7285bf13b2a6" containerName="payload-processing" containerID="cri-o://fc87680d6783c806d0c7a6ceca2751965edcda83e0dc4b2b3645b6991fac2721" gracePeriod=30
Oct 30 13:06:21.890686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:21.880750    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"]
Oct 30 13:06:21.890686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:21.881012    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc" podUID="db2571c8-8d07-4569-a446-43c61d8632cb" containerName="payload-pre-processing" containerID="cri-o://177eff8b0ff94d5d00afef8312c8f14827f8d61c051189d54875269debdaabcb" gracePeriod=30
Oct 30 13:06:22.302384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.294107    4429 generic.go:356] "Generic (PLEG): container finished" podID="db2571c8-8d07-4569-a446-43c61d8632cb" containerID="177eff8b0ff94d5d00afef8312c8f14827f8d61c051189d54875269debdaabcb" exitCode=0
Oct 30 13:06:22.302384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.294341    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc" event={"ID":"db2571c8-8d07-4569-a446-43c61d8632cb","Type":"ContainerDied","Data":"177eff8b0ff94d5d00afef8312c8f14827f8d61c051189d54875269debdaabcb"}
Oct 30 13:06:22.302384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.297970    4429 generic.go:356] "Generic (PLEG): container finished" podID="693983ad-c737-45cd-9924-7285bf13b2a6" containerID="fc87680d6783c806d0c7a6ceca2751965edcda83e0dc4b2b3645b6991fac2721" exitCode=0
Oct 30 13:06:22.302384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.298186    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn" event={"ID":"693983ad-c737-45cd-9924-7285bf13b2a6","Type":"ContainerDied","Data":"fc87680d6783c806d0c7a6ceca2751965edcda83e0dc4b2b3645b6991fac2721"}
Oct 30 13:06:22.302384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.298240    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn" event={"ID":"693983ad-c737-45cd-9924-7285bf13b2a6","Type":"ContainerDied","Data":"9c636e698128ad0346f0ce918ef745ca131ac7061ed1c98344dbd42793745287"}
Oct 30 13:06:22.302384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.298266    4429 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c636e698128ad0346f0ce918ef745ca131ac7061ed1c98344dbd42793745287"
Oct 30 13:06:22.356868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.356844    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:06:22.361939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.361918    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:06:22.431606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.431525    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2571c8-8d07-4569-a446-43c61d8632cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2571c8-8d07-4569-a446-43c61d8632cb-plugins-config-volume\") pod \"db2571c8-8d07-4569-a446-43c61d8632cb\" (UID: \"db2571c8-8d07-4569-a446-43c61d8632cb\") "
Oct 30 13:06:22.431606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.431574    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2571c8-8d07-4569-a446-43c61d8632cb-kube-api-access-q29mz\" (UniqueName: \"kubernetes.io/projected/db2571c8-8d07-4569-a446-43c61d8632cb-kube-api-access-q29mz\") pod \"db2571c8-8d07-4569-a446-43c61d8632cb\" (UID: \"db2571c8-8d07-4569-a446-43c61d8632cb\") "
Oct 30 13:06:22.432331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.431698    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693983ad-c737-45cd-9924-7285bf13b2a6-kube-api-access-r8wd5\" (UniqueName: \"kubernetes.io/projected/693983ad-c737-45cd-9924-7285bf13b2a6-kube-api-access-r8wd5\") pod \"693983ad-c737-45cd-9924-7285bf13b2a6\" (UID: \"693983ad-c737-45cd-9924-7285bf13b2a6\") "
Oct 30 13:06:22.432331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.431792    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693983ad-c737-45cd-9924-7285bf13b2a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693983ad-c737-45cd-9924-7285bf13b2a6-plugins-config-volume\") pod \"693983ad-c737-45cd-9924-7285bf13b2a6\" (UID: \"693983ad-c737-45cd-9924-7285bf13b2a6\") "
Oct 30 13:06:22.432331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.432228    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693983ad-c737-45cd-9924-7285bf13b2a6-plugins-config-volume" pod "693983ad-c737-45cd-9924-7285bf13b2a6" (UID: "693983ad-c737-45cd-9924-7285bf13b2a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:22.432512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.432342    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2571c8-8d07-4569-a446-43c61d8632cb-plugins-config-volume" pod "db2571c8-8d07-4569-a446-43c61d8632cb" (UID: "db2571c8-8d07-4569-a446-43c61d8632cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:22.432512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.432443    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/693983ad-c737-45cd-9924-7285bf13b2a6-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:22.436125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.436037    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693983ad-c737-45cd-9924-7285bf13b2a6-kube-api-access-r8wd5" pod "693983ad-c737-45cd-9924-7285bf13b2a6" (UID: "693983ad-c737-45cd-9924-7285bf13b2a6"). InnerVolumeSpecName "kube-api-access-r8wd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:22.437761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.437738    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2571c8-8d07-4569-a446-43c61d8632cb-kube-api-access-q29mz" pod "db2571c8-8d07-4569-a446-43c61d8632cb" (UID: "db2571c8-8d07-4569-a446-43c61d8632cb"). InnerVolumeSpecName "kube-api-access-q29mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:22.534245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.534213    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2571c8-8d07-4569-a446-43c61d8632cb-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:22.534245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.534238    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q29mz\" (UniqueName: \"kubernetes.io/projected/db2571c8-8d07-4569-a446-43c61d8632cb-kube-api-access-q29mz\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:22.534245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.534248    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8wd5\" (UniqueName: \"kubernetes.io/projected/693983ad-c737-45cd-9924-7285bf13b2a6-kube-api-access-r8wd5\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:22.880084 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:22.880050    4429 scope.go:122] "RemoveContainer" containerID="44fb3f7da4cccbfc2a6b92dd9aa0877b494cabd88a7cfe9dfae112667bf26b18"
Oct 30 13:06:23.139797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.139723    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:06:23.306089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.306061    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"
Oct 30 13:06:23.306089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.306078    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"
Oct 30 13:06:23.306371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.306095    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc" event={"ID":"db2571c8-8d07-4569-a446-43c61d8632cb","Type":"ContainerDied","Data":"f3a89a13632383bcaebf1b3152b2af6290cf589d70c739331d252b05b623fb83"}
Oct 30 13:06:23.306371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.306155    4429 scope.go:122] "RemoveContainer" containerID="177eff8b0ff94d5d00afef8312c8f14827f8d61c051189d54875269debdaabcb"
Oct 30 13:06:23.366853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.366823    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"]
Oct 30 13:06:23.370088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.370064    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1ee32-84d5d647d7-xdfwc"]
Oct 30 13:06:23.433376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.417543    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"]
Oct 30 13:06:23.433376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:23.422242    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1ee32-59995768db-lhgvn"]
Oct 30 13:06:24.075921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:24.075888    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="693983ad-c737-45cd-9924-7285bf13b2a6" path="/var/lib/kubelet/pods/693983ad-c737-45cd-9924-7285bf13b2a6/volumes"
Oct 30 13:06:24.076331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:24.076302    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2571c8-8d07-4569-a446-43c61d8632cb" path="/var/lib/kubelet/pods/db2571c8-8d07-4569-a446-43c61d8632cb/volumes"
Oct 30 13:06:25.375263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:25.375233    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"]
Oct 30 13:06:25.375664 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:25.375447    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy" containerID="cri-o://21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb" gracePeriod=30
Oct 30 13:06:29.370453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.370422    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"]
Oct 30 13:06:29.371047 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371032    4429 cpu_manager.go:414] "containerMap: removing container" podUID="693983ad-c737-45cd-9924-7285bf13b2a6" containerName="payload-processing"
Oct 30 13:06:29.371095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371048    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693983ad-c737-45cd-9924-7285bf13b2a6" containerName="payload-processing"
Oct 30 13:06:29.371133 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371099    4429 cpu_manager.go:414] "containerMap: removing container" podUID="db2571c8-8d07-4569-a446-43c61d8632cb" containerName="payload-pre-processing"
Oct 30 13:06:29.371133 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371108    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2571c8-8d07-4569-a446-43c61d8632cb" containerName="payload-pre-processing"
Oct 30 13:06:29.371133 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371123    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0116c41d-da7f-4cf9-9c19-91bf810ffc70" containerName="maas-api"
Oct 30 13:06:29.371133 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371130    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0116c41d-da7f-4cf9-9c19-91bf810ffc70" containerName="maas-api"
Oct 30 13:06:29.371311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371270    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0116c41d-da7f-4cf9-9c19-91bf810ffc70" containerName="maas-api"
Oct 30 13:06:29.371311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371305    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="693983ad-c737-45cd-9924-7285bf13b2a6" containerName="payload-processing"
Oct 30 13:06:29.371391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.371317    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2571c8-8d07-4569-a446-43c61d8632cb" containerName="payload-pre-processing"
Oct 30 13:06:29.377251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.377230    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:29.396820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.396784    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"]
Oct 30 13:06:29.406937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.406909    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6e0e74-dockercfg-lvwgn\""
Oct 30 13:06:29.407119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.406974    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6e0e74\""
Oct 30 13:06:29.523307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.508510    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpw9p\" (UniqueName: \"kubernetes.io/projected/97f64658-e0e1-4d57-8e53-ce043bd0dd68-kube-api-access-gpw9p\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh\" (UID: \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:29.535351 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.523793    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f64658-e0e1-4d57-8e53-ce043bd0dd68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh\" (UID: \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:29.637409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.637320    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpw9p\" (UniqueName: \"kubernetes.io/projected/97f64658-e0e1-4d57-8e53-ce043bd0dd68-kube-api-access-gpw9p\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh\" (UID: \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:29.637559 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.637453    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f64658-e0e1-4d57-8e53-ce043bd0dd68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh\" (UID: \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:29.638131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.638112    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f64658-e0e1-4d57-8e53-ce043bd0dd68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh\" (UID: \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:29.662701 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.662669    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpw9p\" (UniqueName: \"kubernetes.io/projected/97f64658-e0e1-4d57-8e53-ce043bd0dd68-kube-api-access-gpw9p\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh\" (UID: \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:29.691886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.691858    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:29.997733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:29.997697    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"]
Oct 30 13:06:29.998594 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:06:29.998559    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f64658_e0e1_4d57_8e53_ce043bd0dd68.slice/crio-0d8d10a2616d7890494118bc3bfa61612027a9a176436cb366517d2d35908336 WatchSource:0}: Error finding container 0d8d10a2616d7890494118bc3bfa61612027a9a176436cb366517d2d35908336: Status 404 returned error can't find the container with id 0d8d10a2616d7890494118bc3bfa61612027a9a176436cb366517d2d35908336
Oct 30 13:06:30.354526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.354444    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh" event={"ID":"97f64658-e0e1-4d57-8e53-ce043bd0dd68","Type":"ContainerStarted","Data":"96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e"}
Oct 30 13:06:30.354526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.354499    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh" event={"ID":"97f64658-e0e1-4d57-8e53-ce043bd0dd68","Type":"ContainerStarted","Data":"0d8d10a2616d7890494118bc3bfa61612027a9a176436cb366517d2d35908336"}
Oct 30 13:06:30.354713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.354544    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:30.401295 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.401231    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh" podStartSLOduration=1.401216732 podStartE2EDuration="1.401216732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:30.400854588 +0000 UTC m=+2048.456348413" watchObservedRunningTime="2026-10-30 13:06:30.401216732 +0000 UTC m=+2048.456710557"
Oct 30 13:06:30.721445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.715906    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:06:30.752783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.752749    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-data\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.752820    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-socket\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.752903    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-kube-api-access-k27xt\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-kube-api-access-k27xt\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753116 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.752998    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-envoy\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753116 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753066    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-podinfo\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753080    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-data" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.753222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753103    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-credential-socket\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753149    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-socket" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.753222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753192    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-token\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-token\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753228    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-certs\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753346    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51866e4a-2e91-47b3-bd71-99ca894f27d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51866e4a-2e91-47b3-bd71-99ca894f27d9-istiod-ca-cert\") pod \"51866e4a-2e91-47b3-bd71-99ca894f27d9\" (UID: \"51866e4a-2e91-47b3-bd71-99ca894f27d9\") "
Oct 30 13:06:30.753460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753411    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-credential-socket" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.753677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753548    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-certs" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.753841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.753808    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51866e4a-2e91-47b3-bd71-99ca894f27d9-istiod-ca-cert" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:30.754327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.754239    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.754432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.754333    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.754432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.754348    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51866e4a-2e91-47b3-bd71-99ca894f27d9-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.754432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.754361    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.754432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.754374    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.757751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.757730    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-token" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:30.757881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.757838    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-podinfo" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:30.757983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.757961    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-kube-api-access-k27xt" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "kube-api-access-k27xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:30.758784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.758761    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-envoy" pod "51866e4a-2e91-47b3-bd71-99ca894f27d9" (UID: "51866e4a-2e91-47b3-bd71-99ca894f27d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:30.855810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.855780    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.855810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.855806    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.855810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.855816    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27xt\" (UniqueName: \"kubernetes.io/projected/51866e4a-2e91-47b3-bd71-99ca894f27d9-kube-api-access-k27xt\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:30.856034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:30.855828    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51866e4a-2e91-47b3-bd71-99ca894f27d9-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:31.362593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.362549    4429 generic.go:356] "Generic (PLEG): container finished" podID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerID="21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb" exitCode=0
Oct 30 13:06:31.362593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.362614    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"
Oct 30 13:06:31.362889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.362630    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" event={"ID":"51866e4a-2e91-47b3-bd71-99ca894f27d9","Type":"ContainerDied","Data":"21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb"}
Oct 30 13:06:31.362889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.362711    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7" event={"ID":"51866e4a-2e91-47b3-bd71-99ca894f27d9","Type":"ContainerDied","Data":"c5794fd4337be976c22ed1dc19631f6cdd937c0e340487fc6eb35209d0c08613"}
Oct 30 13:06:31.362889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.362748    4429 scope.go:122] "RemoveContainer" containerID="21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb"
Oct 30 13:06:31.374157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.374139    4429 scope.go:122] "RemoveContainer" containerID="0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852"
Oct 30 13:06:31.383895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.383877    4429 scope.go:122] "RemoveContainer" containerID="21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb"
Oct 30 13:06:31.384213 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:31.384192    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb\": container with ID starting with 21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb not found: ID does not exist" containerID="21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb"
Oct 30 13:06:31.384343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.384221    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb"} err="failed to get container status \"21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb\": rpc error: code = NotFound desc = could not find container \"21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb\": container with ID starting with 21081984e69c3da45ac4c1f6eb601c6a8150a441ba3b6079a5ff3e948b49c9bb not found: ID does not exist"
Oct 30 13:06:31.384343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.384240    4429 scope.go:122] "RemoveContainer" containerID="0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852"
Oct 30 13:06:31.384548 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:31.384531    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852\": container with ID starting with 0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852 not found: ID does not exist" containerID="0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852"
Oct 30 13:06:31.384584 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.384554    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852"} err="failed to get container status \"0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852\": rpc error: code = NotFound desc = could not find container \"0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852\": container with ID starting with 0c201d0dcfd05bd9985d8dfa09f63dfe5d85b08179fd1784450edb7f4fd5d852 not found: ID does not exist"
Oct 30 13:06:31.426881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.426844    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"]
Oct 30 13:06:31.434799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.434766    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-405eb1-openshift-default-7778848687-nd8k7"]
Oct 30 13:06:31.440372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.440337    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"]
Oct 30 13:06:31.440619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:31.440588    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy" containerID="cri-o://f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b" gracePeriod=30
Oct 30 13:06:32.075741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.075709    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" path="/var/lib/kubelet/pods/51866e4a-2e91-47b3-bd71-99ca894f27d9/volumes"
Oct 30 13:06:32.533483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.533451    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"]
Oct 30 13:06:32.534074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.534059    4429 cpu_manager.go:414] "containerMap: removing container" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy"
Oct 30 13:06:32.534119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.534075    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy"
Oct 30 13:06:32.534119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.534088    4429 cpu_manager.go:414] "containerMap: removing container" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy"
Oct 30 13:06:32.534119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.534093    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy"
Oct 30 13:06:32.534210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.534204    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy"
Oct 30 13:06:32.534244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.534213    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="51866e4a-2e91-47b3-bd71-99ca894f27d9" containerName="istio-proxy"
Oct 30 13:06:32.539752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.539729    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:32.558883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.558850    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"]
Oct 30 13:06:32.568225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.568195    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-76c107\""
Oct 30 13:06:32.568568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.568551    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-76c107-dockercfg-tq8p4\""
Oct 30 13:06:32.578294 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.578253    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c14315b-e7e2-4336-8239-c848ceb18e34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc\" (UID: \"8c14315b-e7e2-4336-8239-c848ceb18e34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:32.578424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.578405    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f86w2\" (UniqueName: \"kubernetes.io/projected/8c14315b-e7e2-4336-8239-c848ceb18e34-kube-api-access-f86w2\") pod \"payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc\" (UID: \"8c14315b-e7e2-4336-8239-c848ceb18e34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:32.684263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.681273    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c14315b-e7e2-4336-8239-c848ceb18e34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc\" (UID: \"8c14315b-e7e2-4336-8239-c848ceb18e34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:32.684263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.681469    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f86w2\" (UniqueName: \"kubernetes.io/projected/8c14315b-e7e2-4336-8239-c848ceb18e34-kube-api-access-f86w2\") pod \"payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc\" (UID: \"8c14315b-e7e2-4336-8239-c848ceb18e34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:32.684263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.683007    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c14315b-e7e2-4336-8239-c848ceb18e34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc\" (UID: \"8c14315b-e7e2-4336-8239-c848ceb18e34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:32.697922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.688087    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"]
Oct 30 13:06:32.701013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.700131    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:32.725095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.703607    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"]
Oct 30 13:06:32.725095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.725014    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f86w2\" (UniqueName: \"kubernetes.io/projected/8c14315b-e7e2-4336-8239-c848ceb18e34-kube-api-access-f86w2\") pod \"payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc\" (UID: \"8c14315b-e7e2-4336-8239-c848ceb18e34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:32.783121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.783082    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4knw\" (UniqueName: \"kubernetes.io/projected/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-kube-api-access-d4knw\") pod \"payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2\" (UID: \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:32.783312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.783170    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2\" (UID: \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:32.855084 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.855015    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:32.884335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.884298    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2\" (UID: \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:32.884604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.884585    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4knw\" (UniqueName: \"kubernetes.io/projected/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-kube-api-access-d4knw\") pod \"payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2\" (UID: \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:32.884976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.884958    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2\" (UID: \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:32.911052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:32.910870    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4knw\" (UniqueName: \"kubernetes.io/projected/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-kube-api-access-d4knw\") pod \"payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2\" (UID: \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:33.070972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.070943    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:33.156877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.156847    4429 patch_prober.go:29] interesting pod/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:06:33.157018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.156903    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:06:33.161379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.161351    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"]
Oct 30 13:06:33.161483 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:06:33.161441    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c14315b_e7e2_4336_8239_c848ceb18e34.slice/crio-588d459a826a7b6686b8fd32a807428214b87008a697bb70750ba508dc0456d3 WatchSource:0}: Error finding container 588d459a826a7b6686b8fd32a807428214b87008a697bb70750ba508dc0456d3: Status 404 returned error can't find the container with id 588d459a826a7b6686b8fd32a807428214b87008a697bb70750ba508dc0456d3
Oct 30 13:06:33.375340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.374477    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"]
Oct 30 13:06:33.375340 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:06:33.374870    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3c28da_1f75_4e0d_bbdc_53b2a3d02f06.slice/crio-d7d0e40b68ab2e017f9d29085379549efd3128c5ffc04a10881616019f65cbba WatchSource:0}: Error finding container d7d0e40b68ab2e017f9d29085379549efd3128c5ffc04a10881616019f65cbba: Status 404 returned error can't find the container with id d7d0e40b68ab2e017f9d29085379549efd3128c5ffc04a10881616019f65cbba
Oct 30 13:06:33.383588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.383536    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc" event={"ID":"8c14315b-e7e2-4336-8239-c848ceb18e34","Type":"ContainerStarted","Data":"27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a"}
Oct 30 13:06:33.383588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.383585    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc" event={"ID":"8c14315b-e7e2-4336-8239-c848ceb18e34","Type":"ContainerStarted","Data":"588d459a826a7b6686b8fd32a807428214b87008a697bb70750ba508dc0456d3"}
Oct 30 13:06:33.383682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.383648    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:33.431276 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.431237    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc" podStartSLOduration=1.43122167 podStartE2EDuration="1.43122167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:33.429835493 +0000 UTC m=+2051.485329318" watchObservedRunningTime="2026-10-30 13:06:33.43122167 +0000 UTC m=+2051.486715496"
Oct 30 13:06:33.482377 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.482341    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"]
Oct 30 13:06:33.487301 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.487267    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.520359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.510556    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"]
Oct 30 13:06:33.586524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.586487    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8c35df-openshift-default-dockercfg-24tn6\""
Oct 30 13:06:33.596806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.596774    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-certs\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.596993 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.596905    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istiod-ca-cert\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.596993 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.596978    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-podinfo\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.597097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.597033    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-credential-socket\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.597250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.597219    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-data\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.597407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.597265    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-token\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.597469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.597416    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-socket\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.597469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.597450    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qjt\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-kube-api-access-l7qjt\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.597574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.597560    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-envoy\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699309    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istiod-ca-cert\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699356    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-podinfo\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699414    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-credential-socket\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699686    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-data\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699729    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-token\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699831    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-socket\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699876    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qjt\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-kube-api-access-l7qjt\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699963    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-envoy\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.699997    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-certs\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.700139    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istiod-ca-cert\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.700379    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-certs\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.700577    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-socket\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.700818    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-credential-socket\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.701011    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-data\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.702765    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-envoy\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.703941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.703397    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-podinfo\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.725634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.725598    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-token\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.725801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.725749    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qjt\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-kube-api-access-l7qjt\") pod \"e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") " pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:33.815505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:33.815472    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:34.150211 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:06:34.150181    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5058c878_b244_4c68_8b2d_b5f7dc0c7391.slice/crio-59eabfc9710b682e234f2674fb5cc5032c1942dc0c57ca1152001c7e18cc9d35 WatchSource:0}: Error finding container 59eabfc9710b682e234f2674fb5cc5032c1942dc0c57ca1152001c7e18cc9d35: Status 404 returned error can't find the container with id 59eabfc9710b682e234f2674fb5cc5032c1942dc0c57ca1152001c7e18cc9d35
Oct 30 13:06:34.151213 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.151187    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"]
Oct 30 13:06:34.153327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.153270    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:34.153423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.153357    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:34.153423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.153385    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:06:34.392234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.392192    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2" event={"ID":"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06","Type":"ContainerStarted","Data":"6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1"}
Oct 30 13:06:34.392444 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.392263    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2" event={"ID":"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06","Type":"ContainerStarted","Data":"d7d0e40b68ab2e017f9d29085379549efd3128c5ffc04a10881616019f65cbba"}
Oct 30 13:06:34.392444 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.392327    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:34.397698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.397625    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" event={"ID":"5058c878-b244-4c68-8b2d-b5f7dc0c7391","Type":"ContainerStarted","Data":"eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026"}
Oct 30 13:06:34.397698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.397666    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" event={"ID":"5058c878-b244-4c68-8b2d-b5f7dc0c7391","Type":"ContainerStarted","Data":"59eabfc9710b682e234f2674fb5cc5032c1942dc0c57ca1152001c7e18cc9d35"}
Oct 30 13:06:34.459632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.459555    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2" podStartSLOduration=2.459536157 podStartE2EDuration="2.459536157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:34.455685239 +0000 UTC m=+2052.511179062" watchObservedRunningTime="2026-10-30 13:06:34.459536157 +0000 UTC m=+2052.515029983"
Oct 30 13:06:34.540320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.524963    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" podStartSLOduration=1.524944245 podStartE2EDuration="1.524944245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:34.524448132 +0000 UTC m=+2052.579941960" watchObservedRunningTime="2026-10-30 13:06:34.524944245 +0000 UTC m=+2052.580438070"
Oct 30 13:06:34.816467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:34.816424    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:35.821955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:35.821925    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:36.409707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.409668    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:36.410698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.410682    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:06:36.731180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.731155    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:06:36.854117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854071    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-certs\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.854576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854155    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-envoy\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.854576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854197    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-kube-api-access-4w4qq\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-kube-api-access-4w4qq\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.854576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854394    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657cfe5f-d9ae-42da-8649-e7de001439cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657cfe5f-d9ae-42da-8649-e7de001439cd-istiod-ca-cert\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.854576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854460    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-socket\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.854576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854508    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-credential-socket\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.854762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854601    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-data\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.854762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854635    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-podinfo\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.854762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.854676    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-token\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-token\") pod \"657cfe5f-d9ae-42da-8649-e7de001439cd\" (UID: \"657cfe5f-d9ae-42da-8649-e7de001439cd\") "
Oct 30 13:06:36.856112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.856082    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-credential-socket" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.857383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.856200    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-socket" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.857383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.856354    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-data" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.857383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.856599    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-certs" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.857383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.856712    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657cfe5f-d9ae-42da-8649-e7de001439cd-istiod-ca-cert" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:36.859487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.859411    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-envoy" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:36.859745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.859727    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-podinfo" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:06:36.859837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.859782    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-token" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:36.863491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.863462    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-kube-api-access-4w4qq" pod "657cfe5f-d9ae-42da-8649-e7de001439cd" (UID: "657cfe5f-d9ae-42da-8649-e7de001439cd"). InnerVolumeSpecName "kube-api-access-4w4qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:36.957055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.956982    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.957055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.957009    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.957055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.957019    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4qq\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-kube-api-access-4w4qq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.957055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.957028    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657cfe5f-d9ae-42da-8649-e7de001439cd-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.957055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.957036    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.957055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.957044    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.957055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.957052    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.957055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.957060    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:36.957429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:36.957067    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657cfe5f-d9ae-42da-8649-e7de001439cd-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:37.417562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.417475    4429 generic.go:356] "Generic (PLEG): container finished" podID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerID="f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b" exitCode=0
Oct 30 13:06:37.417745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.417558    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"
Oct 30 13:06:37.417745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.417575    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" event={"ID":"657cfe5f-d9ae-42da-8649-e7de001439cd","Type":"ContainerDied","Data":"f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b"}
Oct 30 13:06:37.417745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.417633    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7" event={"ID":"657cfe5f-d9ae-42da-8649-e7de001439cd","Type":"ContainerDied","Data":"17e937c9ccc40303f52b336f71e9283ceb422da7b182a44bbe4cef41dbe8a873"}
Oct 30 13:06:37.417745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.417660    4429 scope.go:122] "RemoveContainer" containerID="f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b"
Oct 30 13:06:37.431995 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.431975    4429 scope.go:122] "RemoveContainer" containerID="85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86"
Oct 30 13:06:37.441556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.441537    4429 scope.go:122] "RemoveContainer" containerID="f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b"
Oct 30 13:06:37.441828 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:37.441809    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b\": container with ID starting with f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b not found: ID does not exist" containerID="f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b"
Oct 30 13:06:37.441907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.441836    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b"} err="failed to get container status \"f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b\": rpc error: code = NotFound desc = could not find container \"f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b\": container with ID starting with f0b858ca8f1703ab773eed71c37d5b15307a785e8b5fc739e2127955c331368b not found: ID does not exist"
Oct 30 13:06:37.441907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.441853    4429 scope.go:122] "RemoveContainer" containerID="85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86"
Oct 30 13:06:37.442148 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:37.442130    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86\": container with ID starting with 85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86 not found: ID does not exist" containerID="85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86"
Oct 30 13:06:37.442196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.442155    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86"} err="failed to get container status \"85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86\": rpc error: code = NotFound desc = could not find container \"85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86\": container with ID starting with 85fe91842b194c768f74b404fe3e42d6994933fff205e8743d4f93e13273bd86 not found: ID does not exist"
Oct 30 13:06:37.476601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.476580    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"]
Oct 30 13:06:37.490639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:37.490619    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1ee32-openshift-default-85d4d9cf9f-2nhn7"]
Oct 30 13:06:38.077206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.077175    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" path="/var/lib/kubelet/pods/657cfe5f-d9ae-42da-8649-e7de001439cd/volumes"
Oct 30 13:06:38.842616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.842586    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"]
Oct 30 13:06:38.843189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.843175    4429 cpu_manager.go:414] "containerMap: removing container" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy"
Oct 30 13:06:38.843233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.843190    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy"
Oct 30 13:06:38.843333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.843321    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy"
Oct 30 13:06:38.843377 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.843341    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy"
Oct 30 13:06:38.848259 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.848232    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:38.868631 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.868604    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"]
Oct 30 13:06:38.881011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.880972    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8c35df\""
Oct 30 13:06:38.881130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.881004    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8c35df-dockercfg-xtr67\""
Oct 30 13:06:38.965662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.965624    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"]
Oct 30 13:06:38.966856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.966827    4429 cpu_manager.go:414] "containerMap: removing container" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy"
Oct 30 13:06:38.967023 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.967009    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657cfe5f-d9ae-42da-8649-e7de001439cd" containerName="istio-proxy"
Oct 30 13:06:38.971440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.971421    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:38.985825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.985795    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707e9065-0497-4d0a-a4b4-8241ea0546e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6\" (UID: \"707e9065-0497-4d0a-a4b4-8241ea0546e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:38.990223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:38.990200    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhmv\" (UniqueName: \"kubernetes.io/projected/707e9065-0497-4d0a-a4b4-8241ea0546e2-kube-api-access-rrhmv\") pod \"payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6\" (UID: \"707e9065-0497-4d0a-a4b4-8241ea0546e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:39.033778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.033744    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"]
Oct 30 13:06:39.092309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.092264    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935a4c8-fd70-4e86-833f-a6d5babf2f91-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg\" (UID: \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:39.092880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.092334    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjggx\" (UniqueName: \"kubernetes.io/projected/e935a4c8-fd70-4e86-833f-a6d5babf2f91-kube-api-access-gjggx\") pod \"payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg\" (UID: \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:39.092880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.092427    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707e9065-0497-4d0a-a4b4-8241ea0546e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6\" (UID: \"707e9065-0497-4d0a-a4b4-8241ea0546e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:39.092880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.092496    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhmv\" (UniqueName: \"kubernetes.io/projected/707e9065-0497-4d0a-a4b4-8241ea0546e2-kube-api-access-rrhmv\") pod \"payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6\" (UID: \"707e9065-0497-4d0a-a4b4-8241ea0546e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:39.093332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.093306    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707e9065-0497-4d0a-a4b4-8241ea0546e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6\" (UID: \"707e9065-0497-4d0a-a4b4-8241ea0546e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:39.118706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.118684    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhmv\" (UniqueName: \"kubernetes.io/projected/707e9065-0497-4d0a-a4b4-8241ea0546e2-kube-api-access-rrhmv\") pod \"payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6\" (UID: \"707e9065-0497-4d0a-a4b4-8241ea0546e2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:39.144501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.144473    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"]
Oct 30 13:06:39.161643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.161621    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:39.174586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.174561    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"]
Oct 30 13:06:39.193967 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.193930    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935a4c8-fd70-4e86-833f-a6d5babf2f91-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg\" (UID: \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:39.194090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.194010    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjggx\" (UniqueName: \"kubernetes.io/projected/e935a4c8-fd70-4e86-833f-a6d5babf2f91-kube-api-access-gjggx\") pod \"payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg\" (UID: \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:39.194670 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.194652    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935a4c8-fd70-4e86-833f-a6d5babf2f91-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg\" (UID: \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:39.219739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.219713    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjggx\" (UniqueName: \"kubernetes.io/projected/e935a4c8-fd70-4e86-833f-a6d5babf2f91-kube-api-access-gjggx\") pod \"payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg\" (UID: \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:39.337999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.337974    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:39.473741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.473707    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"]
Oct 30 13:06:39.657338 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:06:39.657308    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode935a4c8_fd70_4e86_833f_a6d5babf2f91.slice/crio-f4c711117cafe7a2de5ddf6e29b88a07b9d47a33e0b80011cf128be414131a10 WatchSource:0}: Error finding container f4c711117cafe7a2de5ddf6e29b88a07b9d47a33e0b80011cf128be414131a10: Status 404 returned error can't find the container with id f4c711117cafe7a2de5ddf6e29b88a07b9d47a33e0b80011cf128be414131a10
Oct 30 13:06:39.658166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:39.658109    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"]
Oct 30 13:06:40.457924 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:40.457885    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6" event={"ID":"707e9065-0497-4d0a-a4b4-8241ea0546e2","Type":"ContainerStarted","Data":"083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3"}
Oct 30 13:06:40.458430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:40.457953    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6" event={"ID":"707e9065-0497-4d0a-a4b4-8241ea0546e2","Type":"ContainerStarted","Data":"9f9e3268ad6001851158f436c90c4ea87bcdab9efe23b60f53097caee6e8e45d"}
Oct 30 13:06:40.458430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:40.458062    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:40.459553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:40.459527    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg" event={"ID":"e935a4c8-fd70-4e86-833f-a6d5babf2f91","Type":"ContainerStarted","Data":"9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2"}
Oct 30 13:06:40.459680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:40.459571    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg" event={"ID":"e935a4c8-fd70-4e86-833f-a6d5babf2f91","Type":"ContainerStarted","Data":"f4c711117cafe7a2de5ddf6e29b88a07b9d47a33e0b80011cf128be414131a10"}
Oct 30 13:06:40.459680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:40.459613    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:40.504222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:40.504164    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6" podStartSLOduration=2.504148419 podStartE2EDuration="2.504148419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:40.501237409 +0000 UTC m=+2058.556731234" watchObservedRunningTime="2026-10-30 13:06:40.504148419 +0000 UTC m=+2058.559642246"
Oct 30 13:06:40.549068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:40.549021    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg" podStartSLOduration=2.549010542 podStartE2EDuration="2.549010542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:40.545802949 +0000 UTC m=+2058.601296771" watchObservedRunningTime="2026-10-30 13:06:40.549010542 +0000 UTC m=+2058.604504367"
Oct 30 13:06:41.365849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:41.365818    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:06:42.381331 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:42.381270    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:06:43.655028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.654998    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"]
Oct 30 13:06:43.659449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.659429    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.682904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.682874    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6e0e74-serving-cert\""
Oct 30 13:06:43.704350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.704322    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"]
Oct 30 13:06:43.768251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.768222    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-metrics-certs\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.768428 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.768363    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0917c4df-8abd-4e0f-837d-254197edfb36-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.768501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.768460    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnc5\" (UniqueName: \"kubernetes.io/projected/0917c4df-8abd-4e0f-837d-254197edfb36-kube-api-access-rwnc5\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.768561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.768545    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.870491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.870450    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.870692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.870573    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-metrics-certs\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.870692 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:43.870612    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-serving-cert: secret "maas-api-e2e-ipp-b-6e0e74-serving-cert" not found
Oct 30 13:06:43.870807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.870689    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0917c4df-8abd-4e0f-837d-254197edfb36-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.870807 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:43.870706    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls podName:0917c4df-8abd-4e0f-837d-254197edfb36 nodeName:}" failed. No retries permitted until 2026-10-30 13:06:44.37068518 +0000 UTC m=+2062.426178998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls") pod "maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz" (UID: "0917c4df-8abd-4e0f-837d-254197edfb36") : secret "maas-api-e2e-ipp-b-6e0e74-serving-cert" not found
Oct 30 13:06:43.870923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.870891    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnc5\" (UniqueName: \"kubernetes.io/projected/0917c4df-8abd-4e0f-837d-254197edfb36-kube-api-access-rwnc5\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.871557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.871532    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0917c4df-8abd-4e0f-837d-254197edfb36-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.873507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.873485    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-metrics-certs\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:43.898753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:43.898720    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnc5\" (UniqueName: \"kubernetes.io/projected/0917c4df-8abd-4e0f-837d-254197edfb36-kube-api-access-rwnc5\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:44.145755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.145712    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk" podUID="345c221d-1715-4a01-aee5-92b5be2cf96d" containerName="payload-processing" containerID="cri-o://d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0" gracePeriod=25
Oct 30 13:06:44.175013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.174983    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm" podUID="2f91a6a6-9720-4e68-9cb0-e8424ed55f70" containerName="payload-pre-processing" containerID="cri-o://ffc199e19d3b0c943fb817bd7d0736c551c301ca9aad0d2dac3b03f39e90d49d" gracePeriod=25
Oct 30 13:06:44.380793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.380752    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:44.414726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.413721    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:44.416948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.416534    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:06:44.441473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.441430    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:06:44.491866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.491834    4429 generic.go:356] "Generic (PLEG): container finished" podID="2f91a6a6-9720-4e68-9cb0-e8424ed55f70" containerID="ffc199e19d3b0c943fb817bd7d0736c551c301ca9aad0d2dac3b03f39e90d49d" exitCode=0
Oct 30 13:06:44.492037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.492019    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm" event={"ID":"2f91a6a6-9720-4e68-9cb0-e8424ed55f70","Type":"ContainerDied","Data":"ffc199e19d3b0c943fb817bd7d0736c551c301ca9aad0d2dac3b03f39e90d49d"}
Oct 30 13:06:44.493849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.493824    4429 generic.go:356] "Generic (PLEG): container finished" podID="345c221d-1715-4a01-aee5-92b5be2cf96d" containerID="d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0" exitCode=0
Oct 30 13:06:44.494089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.494073    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk" event={"ID":"345c221d-1715-4a01-aee5-92b5be2cf96d","Type":"ContainerDied","Data":"d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0"}
Oct 30 13:06:44.494203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.494192    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk" event={"ID":"345c221d-1715-4a01-aee5-92b5be2cf96d","Type":"ContainerDied","Data":"889aca43f66fd7bb06f050daa1bfe4314fe5c6404afe0df1d44fb22c10080996"}
Oct 30 13:06:44.494358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.494346    4429 scope.go:122] "RemoveContainer" containerID="d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0"
Oct 30 13:06:44.494607 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.494596    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"
Oct 30 13:06:44.508467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.508446    4429 scope.go:122] "RemoveContainer" containerID="d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0"
Oct 30 13:06:44.508796 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:06:44.508775    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0\": container with ID starting with d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0 not found: ID does not exist" containerID="d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0"
Oct 30 13:06:44.508908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.508801    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0"} err="failed to get container status \"d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0\": rpc error: code = NotFound desc = could not find container \"d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0\": container with ID starting with d7908a5f6c18de5bbdef2c9ad0f16b2691029be2906780650aee4b44f25978c0 not found: ID does not exist"
Oct 30 13:06:44.511390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.511370    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:06:44.575384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.575355    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:44.583999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.582577    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-kube-api-access-kdjj7\" (UniqueName: \"kubernetes.io/projected/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-kube-api-access-kdjj7\") pod \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\" (UID: \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\") "
Oct 30 13:06:44.583999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.582633    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345c221d-1715-4a01-aee5-92b5be2cf96d-kube-api-access-bz25p\" (UniqueName: \"kubernetes.io/projected/345c221d-1715-4a01-aee5-92b5be2cf96d-kube-api-access-bz25p\") pod \"345c221d-1715-4a01-aee5-92b5be2cf96d\" (UID: \"345c221d-1715-4a01-aee5-92b5be2cf96d\") "
Oct 30 13:06:44.583999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.582789    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-plugins-config-volume\") pod \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\" (UID: \"2f91a6a6-9720-4e68-9cb0-e8424ed55f70\") "
Oct 30 13:06:44.583999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.582927    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/345c221d-1715-4a01-aee5-92b5be2cf96d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345c221d-1715-4a01-aee5-92b5be2cf96d-plugins-config-volume\") pod \"345c221d-1715-4a01-aee5-92b5be2cf96d\" (UID: \"345c221d-1715-4a01-aee5-92b5be2cf96d\") "
Oct 30 13:06:44.584625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.584460    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-plugins-config-volume" pod "2f91a6a6-9720-4e68-9cb0-e8424ed55f70" (UID: "2f91a6a6-9720-4e68-9cb0-e8424ed55f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:44.585600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.584998    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345c221d-1715-4a01-aee5-92b5be2cf96d-plugins-config-volume" pod "345c221d-1715-4a01-aee5-92b5be2cf96d" (UID: "345c221d-1715-4a01-aee5-92b5be2cf96d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:06:44.586203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.586175    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345c221d-1715-4a01-aee5-92b5be2cf96d-kube-api-access-bz25p" pod "345c221d-1715-4a01-aee5-92b5be2cf96d" (UID: "345c221d-1715-4a01-aee5-92b5be2cf96d"). InnerVolumeSpecName "kube-api-access-bz25p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:44.587853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.587829    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-kube-api-access-kdjj7" pod "2f91a6a6-9720-4e68-9cb0-e8424ed55f70" (UID: "2f91a6a6-9720-4e68-9cb0-e8424ed55f70"). InnerVolumeSpecName "kube-api-access-kdjj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:44.702770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.702014    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjj7\" (UniqueName: \"kubernetes.io/projected/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-kube-api-access-kdjj7\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:44.702770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.702440    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz25p\" (UniqueName: \"kubernetes.io/projected/345c221d-1715-4a01-aee5-92b5be2cf96d-kube-api-access-bz25p\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:44.702770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.702459    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f91a6a6-9720-4e68-9cb0-e8424ed55f70-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:44.702770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.702472    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/345c221d-1715-4a01-aee5-92b5be2cf96d-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:06:44.855317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.855271    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"]
Oct 30 13:06:44.858207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.858178    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a121dd-646987b794-9bfdk"]
Oct 30 13:06:44.917032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.917008    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"]
Oct 30 13:06:44.917830 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:06:44.917807    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0917c4df_8abd_4e0f_837d_254197edfb36.slice/crio-c4cee6cf8820a0cee505017d2f91f011e927fd1fe616c1608126d5b6185261ff WatchSource:0}: Error finding container c4cee6cf8820a0cee505017d2f91f011e927fd1fe616c1608126d5b6185261ff: Status 404 returned error can't find the container with id c4cee6cf8820a0cee505017d2f91f011e927fd1fe616c1608126d5b6185261ff
Oct 30 13:06:44.919213 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:44.919195    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:06:45.409885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.409859    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:06:45.503494 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.503458    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz" event={"ID":"0917c4df-8abd-4e0f-837d-254197edfb36","Type":"ContainerStarted","Data":"4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073"}
Oct 30 13:06:45.503676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.503527    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz" event={"ID":"0917c4df-8abd-4e0f-837d-254197edfb36","Type":"ContainerStarted","Data":"c4cee6cf8820a0cee505017d2f91f011e927fd1fe616c1608126d5b6185261ff"}
Oct 30 13:06:45.503676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.503559    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:45.507089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.507063    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm" event={"ID":"2f91a6a6-9720-4e68-9cb0-e8424ed55f70","Type":"ContainerDied","Data":"62aaa56e46cdc4f66539e32a7b4c7e4656ff5e5ad62bffa9aec1093334ee036e"}
Oct 30 13:06:45.507190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.507105    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"
Oct 30 13:06:45.507190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.507128    4429 scope.go:122] "RemoveContainer" containerID="ffc199e19d3b0c943fb817bd7d0736c551c301ca9aad0d2dac3b03f39e90d49d"
Oct 30 13:06:45.551341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.551248    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz" podStartSLOduration=2.150419548 podStartE2EDuration="2.551233106s" totalImagesPullingTime="400.813558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:45.550356363 +0000 UTC m=+2063.605850193" watchObservedRunningTime="2026-10-30 13:06:45.551233106 +0000 UTC m=+2063.606726931"
Oct 30 13:06:45.592158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.592060    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"]
Oct 30 13:06:45.595853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:45.595828    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a121dd-5497f8dc87-n2zbm"]
Oct 30 13:06:46.075479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:46.075452    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f91a6a6-9720-4e68-9cb0-e8424ed55f70" path="/var/lib/kubelet/pods/2f91a6a6-9720-4e68-9cb0-e8424ed55f70/volumes"
Oct 30 13:06:46.075859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:46.075844    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345c221d-1715-4a01-aee5-92b5be2cf96d" path="/var/lib/kubelet/pods/345c221d-1715-4a01-aee5-92b5be2cf96d/volumes"
Oct 30 13:06:47.319428 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.319401    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"]
Oct 30 13:06:47.320034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.320020    4429 cpu_manager.go:414] "containerMap: removing container" podUID="345c221d-1715-4a01-aee5-92b5be2cf96d" containerName="payload-processing"
Oct 30 13:06:47.320081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.320036    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345c221d-1715-4a01-aee5-92b5be2cf96d" containerName="payload-processing"
Oct 30 13:06:47.320081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.320050    4429 cpu_manager.go:414] "containerMap: removing container" podUID="2f91a6a6-9720-4e68-9cb0-e8424ed55f70" containerName="payload-pre-processing"
Oct 30 13:06:47.320081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.320055    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f91a6a6-9720-4e68-9cb0-e8424ed55f70" containerName="payload-pre-processing"
Oct 30 13:06:47.320168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.320156    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="345c221d-1715-4a01-aee5-92b5be2cf96d" containerName="payload-processing"
Oct 30 13:06:47.320168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.320165    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f91a6a6-9720-4e68-9cb0-e8424ed55f70" containerName="payload-pre-processing"
Oct 30 13:06:47.324930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.324910    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.353336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.352579    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-76c107-serving-cert\""
Oct 30 13:06:47.370404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.370376    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"]
Oct 30 13:06:47.435733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.435705    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7fnq\" (UniqueName: \"kubernetes.io/projected/97f8256f-4dee-4d1a-91e6-8cfd197c1362-kube-api-access-c7fnq\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.435888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.435760    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f8256f-4dee-4d1a-91e6-8cfd197c1362-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.435888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.435849    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-metrics-certs\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.435997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.435975    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.537333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.537267    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7fnq\" (UniqueName: \"kubernetes.io/projected/97f8256f-4dee-4d1a-91e6-8cfd197c1362-kube-api-access-c7fnq\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.537542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.537387    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f8256f-4dee-4d1a-91e6-8cfd197c1362-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.537542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.537434    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-metrics-certs\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.537667 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.537546    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.543046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.538106    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f8256f-4dee-4d1a-91e6-8cfd197c1362-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.543046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.542580    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-metrics-certs\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.543046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.542792    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.582895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.582828    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7fnq\" (UniqueName: \"kubernetes.io/projected/97f8256f-4dee-4d1a-91e6-8cfd197c1362-kube-api-access-c7fnq\") pod \"maas-api-e2e-ipp-b-76c107-88854f498-v9hx8\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.642735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.642710    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:47.947121 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:06:47.947086    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f8256f_4dee_4d1a_91e6_8cfd197c1362.slice/crio-bb306f5400d1b5097112ca001a84e0c785e937b655a008044ad1e4329bff00ab WatchSource:0}: Error finding container bb306f5400d1b5097112ca001a84e0c785e937b655a008044ad1e4329bff00ab: Status 404 returned error can't find the container with id bb306f5400d1b5097112ca001a84e0c785e937b655a008044ad1e4329bff00ab
Oct 30 13:06:47.947445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:47.947424    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"]
Oct 30 13:06:48.531713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:48.531675    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8" event={"ID":"97f8256f-4dee-4d1a-91e6-8cfd197c1362","Type":"ContainerStarted","Data":"eda4f335c56e792023d4a62203c60c83ad12881acbfab330924cd40ccdbcee57"}
Oct 30 13:06:48.532162 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:48.531741    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8" event={"ID":"97f8256f-4dee-4d1a-91e6-8cfd197c1362","Type":"ContainerStarted","Data":"bb306f5400d1b5097112ca001a84e0c785e937b655a008044ad1e4329bff00ab"}
Oct 30 13:06:48.532162 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:48.531769    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:06:48.591427 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:48.591383    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8" podStartSLOduration=1.145560994 podStartE2EDuration="1.591369262s" totalImagesPullingTime="445.808268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:48.590424432 +0000 UTC m=+2066.645918255" watchObservedRunningTime="2026-10-30 13:06:48.591369262 +0000 UTC m=+2066.646863087"
Oct 30 13:06:51.469936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:51.469900    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:06:51.470478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:51.469990    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:06:51.521024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:51.521001    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:06:54.543393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:06:54.543363    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:07:08.059166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:08.059135    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"]
Oct 30 13:07:11.239466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:11.239424    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"]
Oct 30 13:07:11.284152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:11.284101    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"]
Oct 30 13:07:12.374334 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:07:12.374271    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:07:13.059723 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.059666    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh" podUID="97f64658-e0e1-4d57-8e53-ce043bd0dd68" containerName="payload-pre-processing" containerID="cri-o://96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e" gracePeriod=25
Oct 30 13:07:13.350772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.350752    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:07:13.402457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.402426    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f64658-e0e1-4d57-8e53-ce043bd0dd68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f64658-e0e1-4d57-8e53-ce043bd0dd68-plugins-config-volume\") pod \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\" (UID: \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\") "
Oct 30 13:07:13.402870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.402472    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f64658-e0e1-4d57-8e53-ce043bd0dd68-kube-api-access-gpw9p\" (UniqueName: \"kubernetes.io/projected/97f64658-e0e1-4d57-8e53-ce043bd0dd68-kube-api-access-gpw9p\") pod \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\" (UID: \"97f64658-e0e1-4d57-8e53-ce043bd0dd68\") "
Oct 30 13:07:13.402870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.402758    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f64658-e0e1-4d57-8e53-ce043bd0dd68-plugins-config-volume" pod "97f64658-e0e1-4d57-8e53-ce043bd0dd68" (UID: "97f64658-e0e1-4d57-8e53-ce043bd0dd68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:13.403027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.403011    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f64658-e0e1-4d57-8e53-ce043bd0dd68-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:13.405100 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.405077    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f64658-e0e1-4d57-8e53-ce043bd0dd68-kube-api-access-gpw9p" pod "97f64658-e0e1-4d57-8e53-ce043bd0dd68" (UID: "97f64658-e0e1-4d57-8e53-ce043bd0dd68"). InnerVolumeSpecName "kube-api-access-gpw9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:13.504239 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.504215    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpw9p\" (UniqueName: \"kubernetes.io/projected/97f64658-e0e1-4d57-8e53-ce043bd0dd68-kube-api-access-gpw9p\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:13.694829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.694797    4429 generic.go:356] "Generic (PLEG): container finished" podID="97f64658-e0e1-4d57-8e53-ce043bd0dd68" containerID="96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e" exitCode=0
Oct 30 13:07:13.695015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.694864    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"
Oct 30 13:07:13.695015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.694891    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh" event={"ID":"97f64658-e0e1-4d57-8e53-ce043bd0dd68","Type":"ContainerDied","Data":"96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e"}
Oct 30 13:07:13.695015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.694952    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh" event={"ID":"97f64658-e0e1-4d57-8e53-ce043bd0dd68","Type":"ContainerDied","Data":"0d8d10a2616d7890494118bc3bfa61612027a9a176436cb366517d2d35908336"}
Oct 30 13:07:13.695015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.694974    4429 scope.go:122] "RemoveContainer" containerID="96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e"
Oct 30 13:07:13.705151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.705130    4429 scope.go:122] "RemoveContainer" containerID="96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e"
Oct 30 13:07:13.705444 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:07:13.705426    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e\": container with ID starting with 96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e not found: ID does not exist" containerID="96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e"
Oct 30 13:07:13.705505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.705452    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e"} err="failed to get container status \"96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e\": rpc error: code = NotFound desc = could not find container \"96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e\": container with ID starting with 96e2b7cd5e8d602bb731a3ca9d0ac88963d6b7b72eb567a7e898ad282bb25f3e not found: ID does not exist"
Oct 30 13:07:13.753056 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.753031    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"]
Oct 30 13:07:13.766473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:13.766448    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-698b445bb7-28vdh"]
Oct 30 13:07:14.076358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:14.076278    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f64658-e0e1-4d57-8e53-ce043bd0dd68" path="/var/lib/kubelet/pods/97f64658-e0e1-4d57-8e53-ce043bd0dd68/volumes"
Oct 30 13:07:16.240079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.240043    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2" podUID="ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06" containerName="payload-processing" containerID="cri-o://6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1" gracePeriod=25
Oct 30 13:07:16.284690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.284654    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc" podUID="8c14315b-e7e2-4336-8239-c848ceb18e34" containerName="payload-pre-processing" containerID="cri-o://27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a" gracePeriod=25
Oct 30 13:07:16.571736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.571708    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:07:16.641578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.641550    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-plugins-config-volume\") pod \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\" (UID: \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\") "
Oct 30 13:07:16.641807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.641677    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-kube-api-access-d4knw\" (UniqueName: \"kubernetes.io/projected/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-kube-api-access-d4knw\") pod \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\" (UID: \"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06\") "
Oct 30 13:07:16.642073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.642042    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-plugins-config-volume" pod "ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06" (UID: "ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:16.642418 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.642361    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:16.644481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.644453    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-kube-api-access-d4knw" pod "ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06" (UID: "ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06"). InnerVolumeSpecName "kube-api-access-d4knw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:16.644602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.644576    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:07:16.716053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.716012    4429 generic.go:356] "Generic (PLEG): container finished" podID="ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06" containerID="6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1" exitCode=0
Oct 30 13:07:16.716224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.716068    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"
Oct 30 13:07:16.716224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.716078    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2" event={"ID":"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06","Type":"ContainerDied","Data":"6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1"}
Oct 30 13:07:16.716224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.716132    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2" event={"ID":"ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06","Type":"ContainerDied","Data":"d7d0e40b68ab2e017f9d29085379549efd3128c5ffc04a10881616019f65cbba"}
Oct 30 13:07:16.716224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.716157    4429 scope.go:122] "RemoveContainer" containerID="6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1"
Oct 30 13:07:16.717634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.717611    4429 generic.go:356] "Generic (PLEG): container finished" podID="8c14315b-e7e2-4336-8239-c848ceb18e34" containerID="27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a" exitCode=0
Oct 30 13:07:16.717761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.717656    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"
Oct 30 13:07:16.717761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.717743    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc" event={"ID":"8c14315b-e7e2-4336-8239-c848ceb18e34","Type":"ContainerDied","Data":"27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a"}
Oct 30 13:07:16.717879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.717795    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc" event={"ID":"8c14315b-e7e2-4336-8239-c848ceb18e34","Type":"ContainerDied","Data":"588d459a826a7b6686b8fd32a807428214b87008a697bb70750ba508dc0456d3"}
Oct 30 13:07:16.727723 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.727704    4429 scope.go:122] "RemoveContainer" containerID="6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1"
Oct 30 13:07:16.728017 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:07:16.727998    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1\": container with ID starting with 6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1 not found: ID does not exist" containerID="6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1"
Oct 30 13:07:16.728072 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.728024    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1"} err="failed to get container status \"6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1\": rpc error: code = NotFound desc = could not find container \"6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1\": container with ID starting with 6812a684e2ee9a171ba6b84ff656609101a6b498c06d3f5b06d1a99b552283e1 not found: ID does not exist"
Oct 30 13:07:16.728072 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.728041    4429 scope.go:122] "RemoveContainer" containerID="27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a"
Oct 30 13:07:16.741052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.741036    4429 scope.go:122] "RemoveContainer" containerID="27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a"
Oct 30 13:07:16.741387 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:07:16.741361    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a\": container with ID starting with 27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a not found: ID does not exist" containerID="27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a"
Oct 30 13:07:16.741478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.741395    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a"} err="failed to get container status \"27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a\": rpc error: code = NotFound desc = could not find container \"27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a\": container with ID starting with 27f61f2555dee586d2f68e765fa54cbb90559651a204f002e8b265f1bbe2d56a not found: ID does not exist"
Oct 30 13:07:16.742814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.742796    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c14315b-e7e2-4336-8239-c848ceb18e34-kube-api-access-f86w2\" (UniqueName: \"kubernetes.io/projected/8c14315b-e7e2-4336-8239-c848ceb18e34-kube-api-access-f86w2\") pod \"8c14315b-e7e2-4336-8239-c848ceb18e34\" (UID: \"8c14315b-e7e2-4336-8239-c848ceb18e34\") "
Oct 30 13:07:16.742933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.742917    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c14315b-e7e2-4336-8239-c848ceb18e34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c14315b-e7e2-4336-8239-c848ceb18e34-plugins-config-volume\") pod \"8c14315b-e7e2-4336-8239-c848ceb18e34\" (UID: \"8c14315b-e7e2-4336-8239-c848ceb18e34\") "
Oct 30 13:07:16.743265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.743242    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c14315b-e7e2-4336-8239-c848ceb18e34-plugins-config-volume" pod "8c14315b-e7e2-4336-8239-c848ceb18e34" (UID: "8c14315b-e7e2-4336-8239-c848ceb18e34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:16.743383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.743336    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4knw\" (UniqueName: \"kubernetes.io/projected/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06-kube-api-access-d4knw\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:16.744921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.744904    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c14315b-e7e2-4336-8239-c848ceb18e34-kube-api-access-f86w2" pod "8c14315b-e7e2-4336-8239-c848ceb18e34" (UID: "8c14315b-e7e2-4336-8239-c848ceb18e34"). InnerVolumeSpecName "kube-api-access-f86w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:16.775078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.775053    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"]
Oct 30 13:07:16.784783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.784759    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-76c107-7c5ccc8494-w9bl2"]
Oct 30 13:07:16.844311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.844221    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c14315b-e7e2-4336-8239-c848ceb18e34-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:16.844311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:16.844245    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f86w2\" (UniqueName: \"kubernetes.io/projected/8c14315b-e7e2-4336-8239-c848ceb18e34-kube-api-access-f86w2\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:17.067191 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:17.067168    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"]
Oct 30 13:07:17.080007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:17.079984    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-57c4dbc647-l8pjc"]
Oct 30 13:07:18.079502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.079469    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c14315b-e7e2-4336-8239-c848ceb18e34" path="/var/lib/kubelet/pods/8c14315b-e7e2-4336-8239-c848ceb18e34/volumes"
Oct 30 13:07:18.079896 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.079886    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06" path="/var/lib/kubelet/pods/ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06/volumes"
Oct 30 13:07:18.730858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.730827    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"]
Oct 30 13:07:18.731770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.731746    4429 cpu_manager.go:414] "containerMap: removing container" podUID="97f64658-e0e1-4d57-8e53-ce043bd0dd68" containerName="payload-pre-processing"
Oct 30 13:07:18.731770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.731771    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f64658-e0e1-4d57-8e53-ce043bd0dd68" containerName="payload-pre-processing"
Oct 30 13:07:18.731943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.731794    4429 cpu_manager.go:414] "containerMap: removing container" podUID="8c14315b-e7e2-4336-8239-c848ceb18e34" containerName="payload-pre-processing"
Oct 30 13:07:18.731943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.731804    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c14315b-e7e2-4336-8239-c848ceb18e34" containerName="payload-pre-processing"
Oct 30 13:07:18.731943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.731819    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06" containerName="payload-processing"
Oct 30 13:07:18.731943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.731826    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06" containerName="payload-processing"
Oct 30 13:07:18.732160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.731968    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c14315b-e7e2-4336-8239-c848ceb18e34" containerName="payload-pre-processing"
Oct 30 13:07:18.732160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.731981    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3c28da-1f75-4e0d-bbdc-53b2a3d02f06" containerName="payload-processing"
Oct 30 13:07:18.732160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.732002    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f64658-e0e1-4d57-8e53-ce043bd0dd68" containerName="payload-pre-processing"
Oct 30 13:07:18.737334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.737311    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.755214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.755189    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"]
Oct 30 13:07:18.764822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.764795    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e954cd-openshift-default-dockercfg-b7xk8\""
Oct 30 13:07:18.868974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.868935    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c548aa-4723-4dec-8d9c-15c127040139-istio-podinfo\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.869180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.869106    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-socket\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.869180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.869147    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c548aa-4723-4dec-8d9c-15c127040139-istiod-ca-cert\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.869361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.869213    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r87s\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-kube-api-access-9r87s\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.869361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.869255    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-credential-socket\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.869361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.869355    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-data\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.869590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.869422    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-istio-token\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.869590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.869515    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-envoy\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.869590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.869558    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-certs\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.971603    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-credential-socket\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.971723    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-data\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.971804    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-istio-token\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.971890    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-envoy\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.971933    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-certs\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.971971    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c548aa-4723-4dec-8d9c-15c127040139-istio-podinfo\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.972084    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-socket\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.972114    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c548aa-4723-4dec-8d9c-15c127040139-istiod-ca-cert\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.972187    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r87s\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-kube-api-access-9r87s\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.972807    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-credential-socket\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.973420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.972959    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-data\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.974126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.973698    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-socket\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.974126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.973954    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-certs\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.990355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.976871    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c548aa-4723-4dec-8d9c-15c127040139-istiod-ca-cert\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.990355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.977102    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c548aa-4723-4dec-8d9c-15c127040139-istio-podinfo\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:18.990355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:18.977496    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-envoy\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:19.020716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.001929    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r87s\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-kube-api-access-9r87s\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:19.020716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.017892    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-istio-token\") pod \"e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") " pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:19.069234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.069197    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:19.391845 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:07:19.391818    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c548aa_4723_4dec_8d9c_15c127040139.slice/crio-ae8df21041e4bb1f667c86539af1ebd34834a3da238e049efc3efab769fb0acf WatchSource:0}: Error finding container ae8df21041e4bb1f667c86539af1ebd34834a3da238e049efc3efab769fb0acf: Status 404 returned error can't find the container with id ae8df21041e4bb1f667c86539af1ebd34834a3da238e049efc3efab769fb0acf
Oct 30 13:07:19.392508 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.392484    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"]
Oct 30 13:07:19.394617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.394568    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:19.394716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.394649    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:19.394716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.394687    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:19.743898 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.743861    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" event={"ID":"64c548aa-4723-4dec-8d9c-15c127040139","Type":"ContainerStarted","Data":"002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95"}
Oct 30 13:07:19.744067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.743924    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" event={"ID":"64c548aa-4723-4dec-8d9c-15c127040139","Type":"ContainerStarted","Data":"ae8df21041e4bb1f667c86539af1ebd34834a3da238e049efc3efab769fb0acf"}
Oct 30 13:07:19.876856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:19.876807    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" podStartSLOduration=1.876792698 podStartE2EDuration="1.876792698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:19.862519351 +0000 UTC m=+2097.918013191" watchObservedRunningTime="2026-10-30 13:07:19.876792698 +0000 UTC m=+2097.932286519"
Oct 30 13:07:20.075958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:20.075892    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:20.076096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:20.075970    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:20.750099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:20.750068    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:20.750938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:20.750922    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:07:24.316805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.316768    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"]
Oct 30 13:07:24.322796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.322768    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:24.340782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.340758    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"]
Oct 30 13:07:24.351246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.351228    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e954cd\""
Oct 30 13:07:24.351391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.351369    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e954cd-dockercfg-kmzjj\""
Oct 30 13:07:24.442939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.442908    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"]
Oct 30 13:07:24.448491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.448466    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:24.449877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.449847    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3416cc31-7540-4662-9236-d05f9b7e2510-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d\" (UID: \"3416cc31-7540-4662-9236-d05f9b7e2510\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:24.450125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.450041    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrnd\" (UniqueName: \"kubernetes.io/projected/3416cc31-7540-4662-9236-d05f9b7e2510-kube-api-access-qbrnd\") pod \"payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d\" (UID: \"3416cc31-7540-4662-9236-d05f9b7e2510\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:24.455704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.455681    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"]
Oct 30 13:07:24.551487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.551455    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3416cc31-7540-4662-9236-d05f9b7e2510-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d\" (UID: \"3416cc31-7540-4662-9236-d05f9b7e2510\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:24.551665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.551579    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh\" (UID: \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:24.551665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.551607    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbxh\" (UniqueName: \"kubernetes.io/projected/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-kube-api-access-clbxh\") pod \"payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh\" (UID: \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:24.551665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.551624    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrnd\" (UniqueName: \"kubernetes.io/projected/3416cc31-7540-4662-9236-d05f9b7e2510-kube-api-access-qbrnd\") pod \"payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d\" (UID: \"3416cc31-7540-4662-9236-d05f9b7e2510\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:24.552409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.552386    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3416cc31-7540-4662-9236-d05f9b7e2510-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d\" (UID: \"3416cc31-7540-4662-9236-d05f9b7e2510\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:24.584430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.584356    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrnd\" (UniqueName: \"kubernetes.io/projected/3416cc31-7540-4662-9236-d05f9b7e2510-kube-api-access-qbrnd\") pod \"payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d\" (UID: \"3416cc31-7540-4662-9236-d05f9b7e2510\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:24.636224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.636194    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:24.669232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.669194    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh\" (UID: \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:24.669391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.669262    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbxh\" (UniqueName: \"kubernetes.io/projected/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-kube-api-access-clbxh\") pod \"payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh\" (UID: \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:24.670641 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.670612    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh\" (UID: \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:24.702965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.702941    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbxh\" (UniqueName: \"kubernetes.io/projected/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-kube-api-access-clbxh\") pod \"payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh\" (UID: \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:24.816987 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:24.791115    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:25.013656 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:07:25.013618    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3416cc31_7540_4662_9236_d05f9b7e2510.slice/crio-e3bb37f2d6d9b53b6fc00baded018b5dd82f431ca4a35a8dd10c0cc1aff6bb95 WatchSource:0}: Error finding container e3bb37f2d6d9b53b6fc00baded018b5dd82f431ca4a35a8dd10c0cc1aff6bb95: Status 404 returned error can't find the container with id e3bb37f2d6d9b53b6fc00baded018b5dd82f431ca4a35a8dd10c0cc1aff6bb95
Oct 30 13:07:25.014469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.014447    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"]
Oct 30 13:07:25.183584 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:07:25.183557    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca41b7d_1f5c_4a2e_b892_073a54c4547c.slice/crio-2f98fd4611f432998a26f8ca28cad0eeaeb93c994d90379dfef15097d7ee5c39 WatchSource:0}: Error finding container 2f98fd4611f432998a26f8ca28cad0eeaeb93c994d90379dfef15097d7ee5c39: Status 404 returned error can't find the container with id 2f98fd4611f432998a26f8ca28cad0eeaeb93c994d90379dfef15097d7ee5c39
Oct 30 13:07:25.184183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.184154    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"]
Oct 30 13:07:25.808218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.808182    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d" event={"ID":"3416cc31-7540-4662-9236-d05f9b7e2510","Type":"ContainerStarted","Data":"14a79d39534f411f83cc14ed74885dc278a3d60e4343c758f0f900ebc389d602"}
Oct 30 13:07:25.808703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.808246    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d" event={"ID":"3416cc31-7540-4662-9236-d05f9b7e2510","Type":"ContainerStarted","Data":"e3bb37f2d6d9b53b6fc00baded018b5dd82f431ca4a35a8dd10c0cc1aff6bb95"}
Oct 30 13:07:25.808703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.808408    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:25.809948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.809921    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh" event={"ID":"8ca41b7d-1f5c-4a2e-b892-073a54c4547c","Type":"ContainerStarted","Data":"c08a9f922b5f8ffbee8fc328818cd8e568f9ec82fddd44e9dc9293a7abe4c1d1"}
Oct 30 13:07:25.810064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.809963    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh" event={"ID":"8ca41b7d-1f5c-4a2e-b892-073a54c4547c","Type":"ContainerStarted","Data":"2f98fd4611f432998a26f8ca28cad0eeaeb93c994d90379dfef15097d7ee5c39"}
Oct 30 13:07:25.810064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.809988    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:25.858003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.857951    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d" podStartSLOduration=1.8579403270000001 podStartE2EDuration="1.857940327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:25.855456103 +0000 UTC m=+2103.910949925" watchObservedRunningTime="2026-10-30 13:07:25.857940327 +0000 UTC m=+2103.913434152"
Oct 30 13:07:25.905603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:25.905549    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh" podStartSLOduration=1.905538001 podStartE2EDuration="1.905538001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:25.904188468 +0000 UTC m=+2103.959682290" watchObservedRunningTime="2026-10-30 13:07:25.905538001 +0000 UTC m=+2103.961031823"
Oct 30 13:07:27.700525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.700491    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"]
Oct 30 13:07:27.704917 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.704894    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:27.723374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.723352    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6e0e74-dockercfg-bfk9g\""
Oct 30 13:07:27.733479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.733438    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k68sd\" (UniqueName: \"kubernetes.io/projected/22b2bb3b-68e9-4282-85bc-25b611741d46-kube-api-access-k68sd\") pod \"payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk\" (UID: \"22b2bb3b-68e9-4282-85bc-25b611741d46\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:27.733608 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.733547    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b2bb3b-68e9-4282-85bc-25b611741d46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk\" (UID: \"22b2bb3b-68e9-4282-85bc-25b611741d46\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:27.757310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.743220    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6e0e74\""
Oct 30 13:07:27.757310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.756896    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"]
Oct 30 13:07:27.816251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.816221    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"]
Oct 30 13:07:27.820859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.820838    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:27.835409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.835384    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a724fb-1295-4798-848b-07cdd09344e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk\" (UID: \"79a724fb-1295-4798-848b-07cdd09344e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:27.835538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.835440    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmsq\" (UniqueName: \"kubernetes.io/projected/79a724fb-1295-4798-848b-07cdd09344e5-kube-api-access-djmsq\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk\" (UID: \"79a724fb-1295-4798-848b-07cdd09344e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:27.835538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.835529    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k68sd\" (UniqueName: \"kubernetes.io/projected/22b2bb3b-68e9-4282-85bc-25b611741d46-kube-api-access-k68sd\") pod \"payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk\" (UID: \"22b2bb3b-68e9-4282-85bc-25b611741d46\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:27.835665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.835557    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b2bb3b-68e9-4282-85bc-25b611741d46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk\" (UID: \"22b2bb3b-68e9-4282-85bc-25b611741d46\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:27.836261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.836237    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b2bb3b-68e9-4282-85bc-25b611741d46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk\" (UID: \"22b2bb3b-68e9-4282-85bc-25b611741d46\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:27.846850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.846826    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"]
Oct 30 13:07:27.881223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.881175    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k68sd\" (UniqueName: \"kubernetes.io/projected/22b2bb3b-68e9-4282-85bc-25b611741d46-kube-api-access-k68sd\") pod \"payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk\" (UID: \"22b2bb3b-68e9-4282-85bc-25b611741d46\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:27.937027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.936995    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a724fb-1295-4798-848b-07cdd09344e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk\" (UID: \"79a724fb-1295-4798-848b-07cdd09344e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:27.937227 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.937064    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmsq\" (UniqueName: \"kubernetes.io/projected/79a724fb-1295-4798-848b-07cdd09344e5-kube-api-access-djmsq\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk\" (UID: \"79a724fb-1295-4798-848b-07cdd09344e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:27.937758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.937723    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a724fb-1295-4798-848b-07cdd09344e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk\" (UID: \"79a724fb-1295-4798-848b-07cdd09344e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:27.973359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:27.972458    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmsq\" (UniqueName: \"kubernetes.io/projected/79a724fb-1295-4798-848b-07cdd09344e5-kube-api-access-djmsq\") pod \"payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk\" (UID: \"79a724fb-1295-4798-848b-07cdd09344e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:28.019183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.019152    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:28.138373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.137684    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:28.395495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.394134    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"]
Oct 30 13:07:28.507400 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:07:28.507337    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a724fb_1295_4798_848b_07cdd09344e5.slice/crio-6724f9a13f8d01958c0d6380e9e651d299f48b77869e6808f97716a41ff69c3a WatchSource:0}: Error finding container 6724f9a13f8d01958c0d6380e9e651d299f48b77869e6808f97716a41ff69c3a: Status 404 returned error can't find the container with id 6724f9a13f8d01958c0d6380e9e651d299f48b77869e6808f97716a41ff69c3a
Oct 30 13:07:28.509778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.509755    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"]
Oct 30 13:07:28.844397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.843421    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk" event={"ID":"79a724fb-1295-4798-848b-07cdd09344e5","Type":"ContainerStarted","Data":"b860014add2b2f688ba910ba0e6ad7a5e6b6316dc0f74c2e3b7183cf963f4ed0"}
Oct 30 13:07:28.844397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.843496    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk" event={"ID":"79a724fb-1295-4798-848b-07cdd09344e5","Type":"ContainerStarted","Data":"6724f9a13f8d01958c0d6380e9e651d299f48b77869e6808f97716a41ff69c3a"}
Oct 30 13:07:28.844397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.844378    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:28.846093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.846064    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk" event={"ID":"22b2bb3b-68e9-4282-85bc-25b611741d46","Type":"ContainerStarted","Data":"1ffa970692da2b6e431f7b3eeaf7cf3556c159570e5ee29517c31fa316f585f6"}
Oct 30 13:07:28.846216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.846124    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk" event={"ID":"22b2bb3b-68e9-4282-85bc-25b611741d46","Type":"ContainerStarted","Data":"c8f40a50badb0396357f6ddc73ea488417f39b756ffcdbbeb31f9feb7408f7c7"}
Oct 30 13:07:28.896962 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.896728    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk" podStartSLOduration=1.89671113 podStartE2EDuration="1.89671113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:28.896065179 +0000 UTC m=+2106.951559005" watchObservedRunningTime="2026-10-30 13:07:28.89671113 +0000 UTC m=+2106.952204955"
Oct 30 13:07:28.942912 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:28.942851    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk" podStartSLOduration=1.942839207 podStartE2EDuration="1.942839207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:28.941915316 +0000 UTC m=+2106.997409144" watchObservedRunningTime="2026-10-30 13:07:28.942839207 +0000 UTC m=+2106.998333031"
Oct 30 13:07:29.856872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:29.856803    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:33.098672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.098638    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"]
Oct 30 13:07:33.104042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.104021    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.121024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.121000    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"]
Oct 30 13:07:33.130664 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.130639    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-76c107\""
Oct 30 13:07:33.130863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.130845    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-76c107-dockercfg-d54pb\""
Oct 30 13:07:33.133881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.132697    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188768bd-7283-4c84-b89f-449ce5ae621b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2\" (UID: \"188768bd-7283-4c84-b89f-449ce5ae621b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.133881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.132858    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrds\" (UniqueName: \"kubernetes.io/projected/188768bd-7283-4c84-b89f-449ce5ae621b-kube-api-access-6nrds\") pod \"payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2\" (UID: \"188768bd-7283-4c84-b89f-449ce5ae621b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.285631 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.285543    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188768bd-7283-4c84-b89f-449ce5ae621b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2\" (UID: \"188768bd-7283-4c84-b89f-449ce5ae621b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.285939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.285919    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrds\" (UniqueName: \"kubernetes.io/projected/188768bd-7283-4c84-b89f-449ce5ae621b-kube-api-access-6nrds\") pod \"payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2\" (UID: \"188768bd-7283-4c84-b89f-449ce5ae621b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.287786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.287762    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188768bd-7283-4c84-b89f-449ce5ae621b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2\" (UID: \"188768bd-7283-4c84-b89f-449ce5ae621b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.317414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.315590    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrds\" (UniqueName: \"kubernetes.io/projected/188768bd-7283-4c84-b89f-449ce5ae621b-kube-api-access-6nrds\") pod \"payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2\" (UID: \"188768bd-7283-4c84-b89f-449ce5ae621b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.421211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.421138    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.727750 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:07:33.727718    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188768bd_7283_4c84_b89f_449ce5ae621b.slice/crio-acdb5cdecb2cc31198def3c491ae8b05c9a124ab90784748695de75a6d37fb0b WatchSource:0}: Error finding container acdb5cdecb2cc31198def3c491ae8b05c9a124ab90784748695de75a6d37fb0b: Status 404 returned error can't find the container with id acdb5cdecb2cc31198def3c491ae8b05c9a124ab90784748695de75a6d37fb0b
Oct 30 13:07:33.729124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.729105    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"]
Oct 30 13:07:33.904168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.904135    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2" event={"ID":"188768bd-7283-4c84-b89f-449ce5ae621b","Type":"ContainerStarted","Data":"a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b"}
Oct 30 13:07:33.904333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.904188    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2" event={"ID":"188768bd-7283-4c84-b89f-449ce5ae621b","Type":"ContainerStarted","Data":"acdb5cdecb2cc31198def3c491ae8b05c9a124ab90784748695de75a6d37fb0b"}
Oct 30 13:07:33.904333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.904316    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:33.975697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:33.975589    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2" podStartSLOduration=0.975574389 podStartE2EDuration="975.574389ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:33.963820127 +0000 UTC m=+2112.019313959" watchObservedRunningTime="2026-10-30 13:07:33.975574389 +0000 UTC m=+2112.031068213"
Oct 30 13:07:36.825385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:36.825354    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:07:36.825875 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:36.825609    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:07:37.072068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.070453    4429 patch_prober.go:29] interesting pod/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": read tcp 10.133.0.2:41702->10.133.0.111:15021: read: connection reset by peer" start-of-body=
Oct 30 13:07:37.072068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.070516    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": read tcp 10.133.0.2:41702->10.133.0.111:15021: read: connection reset by peer"
Oct 30 13:07:37.631234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.631204    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"]
Oct 30 13:07:37.631534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.631479    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy" containerID="cri-o://824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510" gracePeriod=30
Oct 30 13:07:37.945820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.945787    4429 generic.go:356] "Generic (PLEG): container finished" podID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerID="eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026" exitCode=0
Oct 30 13:07:37.946244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.945861    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" event={"ID":"5058c878-b244-4c68-8b2d-b5f7dc0c7391","Type":"ContainerDied","Data":"eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026"}
Oct 30 13:07:37.946339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.946321    4429 scope.go:122] "RemoveContainer" containerID="eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026"
Oct 30 13:07:37.947884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.947859    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:37.947975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.947924    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:37.947975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:37.947957    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:07:38.990714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:38.990676    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" event={"ID":"5058c878-b244-4c68-8b2d-b5f7dc0c7391","Type":"ContainerStarted","Data":"6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb"}
Oct 30 13:07:39.816034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:39.816007    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:07:39.816207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:39.816185    4429 patch_prober.go:29] interesting pod/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:07:39.816251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:39.816234    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:07:40.816485 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:40.816453    4429 patch_prober.go:29] interesting pod/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:07:40.816876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:40.816508    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:07:40.881515 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:40.881483    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:07:40.881960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:40.881934    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:07:41.816807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:41.816776    4429 patch_prober.go:29] interesting pod/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:07:41.817236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:41.816825    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:07:42.415344 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:07:42.411660    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:07:42.816561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:42.816526    4429 patch_prober.go:29] interesting pod/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:07:42.816725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:42.816585    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:07:42.981698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:42.981621    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:07:43.026844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.026812    4429 generic.go:356] "Generic (PLEG): container finished" podID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerID="824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510" exitCode=0
Oct 30 13:07:43.027007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.026887    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" event={"ID":"0161fb89-4a52-42c6-a804-e1fcf3d6c40e","Type":"ContainerDied","Data":"824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510"}
Oct 30 13:07:43.027007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.026952    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v" event={"ID":"0161fb89-4a52-42c6-a804-e1fcf3d6c40e","Type":"ContainerDied","Data":"5398bd6da7062e67e2170bbc4004493c09e29f657af53d6d219cf665323fe342"}
Oct 30 13:07:43.027007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.026974    4429 scope.go:122] "RemoveContainer" containerID="824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510"
Oct 30 13:07:43.027108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.027030    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"
Oct 30 13:07:43.040259 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.040240    4429 scope.go:122] "RemoveContainer" containerID="39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2"
Oct 30 13:07:43.083307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.080523    4429 scope.go:122] "RemoveContainer" containerID="824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510"
Oct 30 13:07:43.089180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089137    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-socket\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.089403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089334    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-podinfo\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.089403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089390    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-certs\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.089534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089417    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-envoy\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.089534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089483    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-data\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.089534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089529    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istiod-ca-cert\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.089703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089632    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-credential-socket\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.089762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089728    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-kube-api-access-2h926\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-kube-api-access-2h926\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.089817 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.089774    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-token\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-token\") pod \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\" (UID: \"0161fb89-4a52-42c6-a804-e1fcf3d6c40e\") "
Oct 30 13:07:43.090462 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:07:43.090433    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510\": container with ID starting with 824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510 not found: ID does not exist" containerID="824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510"
Oct 30 13:07:43.090625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.090600    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510"} err="failed to get container status \"824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510\": rpc error: code = NotFound desc = could not find container \"824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510\": container with ID starting with 824cd0f462aa51117c8489f1c4ec6bf814daa6fd8338558251361987665b8510 not found: ID does not exist"
Oct 30 13:07:43.090727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.090717    4429 scope.go:122] "RemoveContainer" containerID="39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2"
Oct 30 13:07:43.091856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.091832    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-certs" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:43.092160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.092142    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-data" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:43.092484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.092465    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-credential-socket" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:43.092813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.092795    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-socket" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:43.093495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.093474    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istiod-ca-cert" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:07:43.093742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.093724    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-token" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:43.095185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.095148    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-envoy" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:07:43.095451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.095424    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-podinfo" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:07:43.096220 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:07:43.096195    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2\": container with ID starting with 39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2 not found: ID does not exist" containerID="39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2"
Oct 30 13:07:43.097495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.097466    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2"} err="failed to get container status \"39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2\": rpc error: code = NotFound desc = could not find container \"39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2\": container with ID starting with 39b6678648a2540fb5f714ab3f4076ae7f53234fba67836af06992e51a93f9b2 not found: ID does not exist"
Oct 30 13:07:43.097617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.096681    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-kube-api-access-2h926" pod "0161fb89-4a52-42c6-a804-e1fcf3d6c40e" (UID: "0161fb89-4a52-42c6-a804-e1fcf3d6c40e"). InnerVolumeSpecName "kube-api-access-2h926". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:07:43.191530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191483    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.191530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191517    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.191530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191532    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.191778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191545    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.191778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191558    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.191778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191572    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.191778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191585    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h926\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-kube-api-access-2h926\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.191778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191599    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.191778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.191613    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0161fb89-4a52-42c6-a804-e1fcf3d6c40e-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:07:43.388476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.388440    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"]
Oct 30 13:07:43.401314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.401275    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a121dd-openshift-default-64755bb657-tbb4v"]
Oct 30 13:07:43.816297 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.816254    4429 patch_prober.go:29] interesting pod/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:07:43.816483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:43.816362    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 13:07:44.083049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:44.082970    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" path="/var/lib/kubelet/pods/0161fb89-4a52-42c6-a804-e1fcf3d6c40e/volumes"
Oct 30 13:07:44.820551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:44.820520    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:07:44.820908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:44.820887    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:07:44.821604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:44.821585    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:07:44.915151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:44.915119    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:07:45.356800 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.356770    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"]
Oct 30 13:07:45.357419 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.357404    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy"
Oct 30 13:07:45.357486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.357421    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy"
Oct 30 13:07:45.357541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.357531    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy"
Oct 30 13:07:45.357575 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.357545    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy"
Oct 30 13:07:45.362653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.362634    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:45.380667 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.380639    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f9ea8f-dockercfg-wb9x9\""
Oct 30 13:07:45.380667 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.380670    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f9ea8f\""
Oct 30 13:07:45.381080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.381064    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"]
Oct 30 13:07:45.563656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.529342    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"]
Oct 30 13:07:45.563656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.530533    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy"
Oct 30 13:07:45.563656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.530555    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0161fb89-4a52-42c6-a804-e1fcf3d6c40e" containerName="istio-proxy"
Oct 30 13:07:45.563656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.534924    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:45.563656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.546816    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9\" (UID: \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:45.563656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.547067    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zsdd\" (UniqueName: \"kubernetes.io/projected/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-kube-api-access-7zsdd\") pod \"payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9\" (UID: \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:45.563656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.547131    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mdps\" (UniqueName: \"kubernetes.io/projected/753a4ac6-b69b-4330-8b36-a11955e43912-kube-api-access-4mdps\") pod \"payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8\" (UID: \"753a4ac6-b69b-4330-8b36-a11955e43912\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:45.563656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.547201    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/753a4ac6-b69b-4330-8b36-a11955e43912-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8\" (UID: \"753a4ac6-b69b-4330-8b36-a11955e43912\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:45.582447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.582408    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"]
Oct 30 13:07:45.651372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.648443    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9\" (UID: \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:45.651372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.648568    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsdd\" (UniqueName: \"kubernetes.io/projected/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-kube-api-access-7zsdd\") pod \"payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9\" (UID: \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:45.651372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.648617    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mdps\" (UniqueName: \"kubernetes.io/projected/753a4ac6-b69b-4330-8b36-a11955e43912-kube-api-access-4mdps\") pod \"payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8\" (UID: \"753a4ac6-b69b-4330-8b36-a11955e43912\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:45.651372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.648683    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/753a4ac6-b69b-4330-8b36-a11955e43912-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8\" (UID: \"753a4ac6-b69b-4330-8b36-a11955e43912\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:45.651372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.649385    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9\" (UID: \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:45.651372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.649545    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/753a4ac6-b69b-4330-8b36-a11955e43912-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8\" (UID: \"753a4ac6-b69b-4330-8b36-a11955e43912\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:45.675178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.675142    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsdd\" (UniqueName: \"kubernetes.io/projected/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-kube-api-access-7zsdd\") pod \"payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9\" (UID: \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:45.675379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.675223    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mdps\" (UniqueName: \"kubernetes.io/projected/753a4ac6-b69b-4330-8b36-a11955e43912-kube-api-access-4mdps\") pod \"payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8\" (UID: \"753a4ac6-b69b-4330-8b36-a11955e43912\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:45.676043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.676026    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:45.781785 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.779337    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"]
Oct 30 13:07:45.806524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.806488    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:45.808066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.808038    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"]
Oct 30 13:07:45.824748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.824714    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e954cd-serving-cert\""
Oct 30 13:07:45.907739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.907649    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:45.970371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.967354    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-metrics-certs\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:45.970371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.967425    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pctkd\" (UniqueName: \"kubernetes.io/projected/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-kube-api-access-pctkd\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:45.970371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.967477    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:45.970371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:45.967535    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.013673 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.013628    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"]
Oct 30 13:07:46.049089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.049057    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9" event={"ID":"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0","Type":"ContainerStarted","Data":"a639f1f7fdc73bbf88f05247b15f571d264b23b5c0e6a5a4dd66d9462d3e2311"}
Oct 30 13:07:46.070968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.070214    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-metrics-certs\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.070968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.070310    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pctkd\" (UniqueName: \"kubernetes.io/projected/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-kube-api-access-pctkd\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.070968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.070350    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.070968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.070399    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.072719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.072692    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.079839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.079799    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.081224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.081189    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-metrics-certs\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.096846 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.096819    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pctkd\" (UniqueName: \"kubernetes.io/projected/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-kube-api-access-pctkd\") pod \"maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.126819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.126780    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:46.243873 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:07:46.243841    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753a4ac6_b69b_4330_8b36_a11955e43912.slice/crio-eaa14ddaf13d013f3b2bd43572e851c3cd3d804fd404977ade772e4db8649858 WatchSource:0}: Error finding container eaa14ddaf13d013f3b2bd43572e851c3cd3d804fd404977ade772e4db8649858: Status 404 returned error can't find the container with id eaa14ddaf13d013f3b2bd43572e851c3cd3d804fd404977ade772e4db8649858
Oct 30 13:07:46.244215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.244191    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"]
Oct 30 13:07:46.430180 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:07:46.430113    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19bb56a_2a1b_4f09_9ac2_fd7354b1824b.slice/crio-6857e776e4399eb795a1798f50404a02d86e8d244d16ba8cb0c1a846a1be80c4 WatchSource:0}: Error finding container 6857e776e4399eb795a1798f50404a02d86e8d244d16ba8cb0c1a846a1be80c4: Status 404 returned error can't find the container with id 6857e776e4399eb795a1798f50404a02d86e8d244d16ba8cb0c1a846a1be80c4
Oct 30 13:07:46.431085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:46.431062    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"]
Oct 30 13:07:47.057783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.057752    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8" event={"ID":"753a4ac6-b69b-4330-8b36-a11955e43912","Type":"ContainerStarted","Data":"9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9"}
Oct 30 13:07:47.057951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.057814    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8" event={"ID":"753a4ac6-b69b-4330-8b36-a11955e43912","Type":"ContainerStarted","Data":"eaa14ddaf13d013f3b2bd43572e851c3cd3d804fd404977ade772e4db8649858"}
Oct 30 13:07:47.058033 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.057972    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:47.059453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.059424    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc" event={"ID":"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b","Type":"ContainerStarted","Data":"c95651700e57c63588fa3be21bfebc03a649f32a932d535266d474330d3ba56c"}
Oct 30 13:07:47.059599 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.059467    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc" event={"ID":"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b","Type":"ContainerStarted","Data":"6857e776e4399eb795a1798f50404a02d86e8d244d16ba8cb0c1a846a1be80c4"}
Oct 30 13:07:47.059599 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.059497    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:47.061022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.061000    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9" event={"ID":"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0","Type":"ContainerStarted","Data":"cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972"}
Oct 30 13:07:47.061110 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.061041    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:07:47.111585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.111522    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8" podStartSLOduration=2.111507404 podStartE2EDuration="2.111507404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:47.110795556 +0000 UTC m=+2125.166289380" watchObservedRunningTime="2026-10-30 13:07:47.111507404 +0000 UTC m=+2125.167001229"
Oct 30 13:07:47.164800 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.164744    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9" podStartSLOduration=2.16472972 podStartE2EDuration="2.16472972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:47.162634648 +0000 UTC m=+2125.218128491" watchObservedRunningTime="2026-10-30 13:07:47.16472972 +0000 UTC m=+2125.220223543"
Oct 30 13:07:47.214333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:47.214210    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc" podStartSLOduration=1.70680605 podStartE2EDuration="2.21419624s" totalImagesPullingTime="507.39019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:47.212412826 +0000 UTC m=+2125.267906650" watchObservedRunningTime="2026-10-30 13:07:47.21419624 +0000 UTC m=+2125.269690064"
Oct 30 13:07:48.779299 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:48.779267    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk_22b2bb3b-68e9-4282-85bc-25b611741d46/payload-processing/0.log"
Oct 30 13:07:50.293673 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.293630    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"]
Oct 30 13:07:50.299976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.298895    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.321386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.320949    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"]
Oct 30 13:07:50.328453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.328427    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-home\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.328602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.328460    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnqx\" (UniqueName: \"kubernetes.io/projected/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-kube-api-access-wfnqx\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.328679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.328602    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tmp-dir\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.328732 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.328680    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-model-cache\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.328921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.328898    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-dshm\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.329011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.328952    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tls-certs\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.329611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.329589    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f9c553\"/\"ipp-route-f9c553-kserve-self-signed-certs\""
Oct 30 13:07:50.329733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.329635    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f9c553\"/\"default-dockercfg-dpsjs\""
Oct 30 13:07:50.331082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.331051    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f9c553\"/\"openshift-service-ca.crt\""
Oct 30 13:07:50.331171 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.331096    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f9c553\"/\"kube-root-ca.crt\""
Oct 30 13:07:50.430224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.430192    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-home\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.430224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.430228    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnqx\" (UniqueName: \"kubernetes.io/projected/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-kube-api-access-wfnqx\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.430504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.430327    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tmp-dir\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.430504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.430387    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-model-cache\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.430623 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.430556    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-dshm\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.430677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.430632    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tls-certs\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.430773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.430747    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-home\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.430773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.430767    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tmp-dir\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.431132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.431104    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-model-cache\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.433247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.433223    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-dshm\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.433371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.433351    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tls-certs\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.455430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.455401    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnqx\" (UniqueName: \"kubernetes.io/projected/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-kube-api-access-wfnqx\") pod \"ipp-route-f9c553-kserve-574485b6bf-rg5tf\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") " pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.627475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.627368    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:50.939043 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:07:50.939008    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95ca8a62_2b26_46d5_8d14_94aadc6ef68b.slice/crio-160733c493b57f9a2cfdab47f3056dbf0b64554e9252895327d03eb3fbbb160a WatchSource:0}: Error finding container 160733c493b57f9a2cfdab47f3056dbf0b64554e9252895327d03eb3fbbb160a: Status 404 returned error can't find the container with id 160733c493b57f9a2cfdab47f3056dbf0b64554e9252895327d03eb3fbbb160a
Oct 30 13:07:50.940438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:50.940415    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"]
Oct 30 13:07:51.088613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:51.088567    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf" event={"ID":"95ca8a62-2b26-46d5-8d14-94aadc6ef68b","Type":"ContainerStarted","Data":"a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6"}
Oct 30 13:07:51.088613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:51.088618    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf" event={"ID":"95ca8a62-2b26-46d5-8d14-94aadc6ef68b","Type":"ContainerStarted","Data":"160733c493b57f9a2cfdab47f3056dbf0b64554e9252895327d03eb3fbbb160a"}
Oct 30 13:07:51.135557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:51.135506    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf" podStartSLOduration=1.135493387 podStartE2EDuration="1.135493387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:51.132572822 +0000 UTC m=+2129.188066643" watchObservedRunningTime="2026-10-30 13:07:51.135493387 +0000 UTC m=+2129.190987211"
Oct 30 13:07:51.627498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:51.627465    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:07:53.072255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:53.072225    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:07:58.077678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:58.077643    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:07:58.078452 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:07:58.077802    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:08:00.628296 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:00.628250    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:08:00.641401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:00.641380    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:08:01.165069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:01.165042    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:08:07.890253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:07.890220    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"]
Oct 30 13:08:07.898521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:07.898482    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:07.937632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:07.937601    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f9ea8f-serving-cert\""
Oct 30 13:08:07.973310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:07.972895    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"]
Oct 30 13:08:07.982267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:07.982220    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9601a9-7990-453b-976e-f952b1686868-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:07.982498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:07.982367    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-metrics-certs\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:07.982498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:07.982488    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjxvc\" (UniqueName: \"kubernetes.io/projected/bc9601a9-7990-453b-976e-f952b1686868-kube-api-access-mjxvc\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:07.982620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:07.982553    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.083663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.083633    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-metrics-certs\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.083835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.083716    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjxvc\" (UniqueName: \"kubernetes.io/projected/bc9601a9-7990-453b-976e-f952b1686868-kube-api-access-mjxvc\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.083835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.083752    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.083930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.083840    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9601a9-7990-453b-976e-f952b1686868-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.084419 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.084397    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9601a9-7990-453b-976e-f952b1686868-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.086332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.086312    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-metrics-certs\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.086427 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.086358    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.144388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.140624    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjxvc\" (UniqueName: \"kubernetes.io/projected/bc9601a9-7990-453b-976e-f952b1686868-kube-api-access-mjxvc\") pod \"maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.215106 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.215068    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:08.596463 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:08:08.596433    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9601a9_7990_453b_976e_f952b1686868.slice/crio-5d04218a5da13247fa2b6101ae39fb7246189cfb1dfd7faa64c5b52481aa7996 WatchSource:0}: Error finding container 5d04218a5da13247fa2b6101ae39fb7246189cfb1dfd7faa64c5b52481aa7996: Status 404 returned error can't find the container with id 5d04218a5da13247fa2b6101ae39fb7246189cfb1dfd7faa64c5b52481aa7996
Oct 30 13:08:08.597481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:08.597446    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"]
Oct 30 13:08:09.211184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:09.211154    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq" event={"ID":"bc9601a9-7990-453b-976e-f952b1686868","Type":"ContainerStarted","Data":"b09c825b88571019904d34b60b91402932d98928d1d116c4b9274a4eda25f9d2"}
Oct 30 13:08:09.211709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:09.211203    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq" event={"ID":"bc9601a9-7990-453b-976e-f952b1686868","Type":"ContainerStarted","Data":"5d04218a5da13247fa2b6101ae39fb7246189cfb1dfd7faa64c5b52481aa7996"}
Oct 30 13:08:09.211709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:09.211376    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:09.284349 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:09.284270    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq" podStartSLOduration=1.967453136 podStartE2EDuration="2.284254787s" totalImagesPullingTime="316.801651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:09.282936222 +0000 UTC m=+2147.338430047" watchObservedRunningTime="2026-10-30 13:08:09.284254787 +0000 UTC m=+2147.339748610"
Oct 30 13:08:12.399943 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:08:12.399905    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:08:15.223362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:15.223334    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:08:17.270909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:17.270815    4429 generic.go:356] "Generic (PLEG): container finished" podID="64c548aa-4723-4dec-8d9c-15c127040139" containerID="002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95" exitCode=0
Oct 30 13:08:17.270909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:17.270890    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" event={"ID":"64c548aa-4723-4dec-8d9c-15c127040139","Type":"ContainerDied","Data":"002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95"}
Oct 30 13:08:17.271473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:17.271418    4429 scope.go:122] "RemoveContainer" containerID="002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95"
Oct 30 13:08:17.273266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:17.273233    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:17.273404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:17.273329    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:17.273404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:17.273365    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:18.282115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:18.282064    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" event={"ID":"64c548aa-4723-4dec-8d9c-15c127040139","Type":"ContainerStarted","Data":"17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4"}
Oct 30 13:08:19.072556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:19.072506    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:08:19.073949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:19.073921    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:08:19.074095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:19.073976    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:08:20.072643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:20.072603    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:08:20.073165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:20.072672    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:08:20.772311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:20.767378    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:08:21.073084 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:21.072737    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:08:21.073084 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:21.072799    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:08:22.069799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:22.069766    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:08:22.070080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:22.070051    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:08:23.073479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:23.073442    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 13:08:23.073916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:23.073506    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 13:08:24.075951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:24.075895    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:08:24.077488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:24.077467    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:08:37.972959 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:37.972921    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"]
Oct 30 13:08:37.991521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:37.991476    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"]
Oct 30 13:08:42.482941 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:08:42.482896    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:08:42.973706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:42.973615    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg" podUID="e935a4c8-fd70-4e86-833f-a6d5babf2f91" containerName="payload-processing" containerID="cri-o://9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2" gracePeriod=25
Oct 30 13:08:42.997582 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:42.992395    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6" podUID="707e9065-0497-4d0a-a4b4-8241ea0546e2" containerName="payload-pre-processing" containerID="cri-o://083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3" gracePeriod=25
Oct 30 13:08:43.289002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.288976    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:08:43.341479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.341445    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:08:43.424662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.424626    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e935a4c8-fd70-4e86-833f-a6d5babf2f91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935a4c8-fd70-4e86-833f-a6d5babf2f91-plugins-config-volume\") pod \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\" (UID: \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\") "
Oct 30 13:08:43.424901 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.424747    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e935a4c8-fd70-4e86-833f-a6d5babf2f91-kube-api-access-gjggx\" (UniqueName: \"kubernetes.io/projected/e935a4c8-fd70-4e86-833f-a6d5babf2f91-kube-api-access-gjggx\") pod \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\" (UID: \"e935a4c8-fd70-4e86-833f-a6d5babf2f91\") "
Oct 30 13:08:43.425097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.425070    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e935a4c8-fd70-4e86-833f-a6d5babf2f91-plugins-config-volume" pod "e935a4c8-fd70-4e86-833f-a6d5babf2f91" (UID: "e935a4c8-fd70-4e86-833f-a6d5babf2f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:43.425350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.425331    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e935a4c8-fd70-4e86-833f-a6d5babf2f91-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:43.427096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.427063    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e935a4c8-fd70-4e86-833f-a6d5babf2f91-kube-api-access-gjggx" pod "e935a4c8-fd70-4e86-833f-a6d5babf2f91" (UID: "e935a4c8-fd70-4e86-833f-a6d5babf2f91"). InnerVolumeSpecName "kube-api-access-gjggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:43.525937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.525839    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707e9065-0497-4d0a-a4b4-8241ea0546e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707e9065-0497-4d0a-a4b4-8241ea0546e2-plugins-config-volume\") pod \"707e9065-0497-4d0a-a4b4-8241ea0546e2\" (UID: \"707e9065-0497-4d0a-a4b4-8241ea0546e2\") "
Oct 30 13:08:43.526449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.526046    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707e9065-0497-4d0a-a4b4-8241ea0546e2-kube-api-access-rrhmv\" (UniqueName: \"kubernetes.io/projected/707e9065-0497-4d0a-a4b4-8241ea0546e2-kube-api-access-rrhmv\") pod \"707e9065-0497-4d0a-a4b4-8241ea0546e2\" (UID: \"707e9065-0497-4d0a-a4b4-8241ea0546e2\") "
Oct 30 13:08:43.526449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.526338    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707e9065-0497-4d0a-a4b4-8241ea0546e2-plugins-config-volume" pod "707e9065-0497-4d0a-a4b4-8241ea0546e2" (UID: "707e9065-0497-4d0a-a4b4-8241ea0546e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:43.526526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.526506    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707e9065-0497-4d0a-a4b4-8241ea0546e2-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:43.526526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.526521    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjggx\" (UniqueName: \"kubernetes.io/projected/e935a4c8-fd70-4e86-833f-a6d5babf2f91-kube-api-access-gjggx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:43.528248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.528218    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707e9065-0497-4d0a-a4b4-8241ea0546e2-kube-api-access-rrhmv" pod "707e9065-0497-4d0a-a4b4-8241ea0546e2" (UID: "707e9065-0497-4d0a-a4b4-8241ea0546e2"). InnerVolumeSpecName "kube-api-access-rrhmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:43.558535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.558503    4429 generic.go:356] "Generic (PLEG): container finished" podID="707e9065-0497-4d0a-a4b4-8241ea0546e2" containerID="083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3" exitCode=0
Oct 30 13:08:43.558727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.558546    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"
Oct 30 13:08:43.558727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.558598    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6" event={"ID":"707e9065-0497-4d0a-a4b4-8241ea0546e2","Type":"ContainerDied","Data":"083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3"}
Oct 30 13:08:43.558727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.558684    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6" event={"ID":"707e9065-0497-4d0a-a4b4-8241ea0546e2","Type":"ContainerDied","Data":"9f9e3268ad6001851158f436c90c4ea87bcdab9efe23b60f53097caee6e8e45d"}
Oct 30 13:08:43.558727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.558716    4429 scope.go:122] "RemoveContainer" containerID="083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3"
Oct 30 13:08:43.560245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.560221    4429 generic.go:356] "Generic (PLEG): container finished" podID="e935a4c8-fd70-4e86-833f-a6d5babf2f91" containerID="9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2" exitCode=0
Oct 30 13:08:43.560461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.560274    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"
Oct 30 13:08:43.560461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.560320    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg" event={"ID":"e935a4c8-fd70-4e86-833f-a6d5babf2f91","Type":"ContainerDied","Data":"9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2"}
Oct 30 13:08:43.560461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.560381    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg" event={"ID":"e935a4c8-fd70-4e86-833f-a6d5babf2f91","Type":"ContainerDied","Data":"f4c711117cafe7a2de5ddf6e29b88a07b9d47a33e0b80011cf128be414131a10"}
Oct 30 13:08:43.570975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.570954    4429 scope.go:122] "RemoveContainer" containerID="083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3"
Oct 30 13:08:43.571372 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:08:43.571350    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3\": container with ID starting with 083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3 not found: ID does not exist" containerID="083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3"
Oct 30 13:08:43.571466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.571382    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3"} err="failed to get container status \"083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3\": rpc error: code = NotFound desc = could not find container \"083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3\": container with ID starting with 083e4179188ad9099f7eb87fe4edf6e027fe4f2ee47b1a0dbd2f026222abb1d3 not found: ID does not exist"
Oct 30 13:08:43.571466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.571407    4429 scope.go:122] "RemoveContainer" containerID="9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2"
Oct 30 13:08:43.581705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.581674    4429 scope.go:122] "RemoveContainer" containerID="9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2"
Oct 30 13:08:43.582014 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:08:43.581996    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2\": container with ID starting with 9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2 not found: ID does not exist" containerID="9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2"
Oct 30 13:08:43.582085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.582021    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2"} err="failed to get container status \"9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2\": rpc error: code = NotFound desc = could not find container \"9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2\": container with ID starting with 9a0c4a9cae3082a1e82888ab053d846628b2e1408b217c39a820da21d25aaea2 not found: ID does not exist"
Oct 30 13:08:43.627614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.627576    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"]
Oct 30 13:08:43.627786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.627689    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrhmv\" (UniqueName: \"kubernetes.io/projected/707e9065-0497-4d0a-a4b4-8241ea0546e2-kube-api-access-rrhmv\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:43.630748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.630726    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8c35df-7944fdb8b6-b8xzg"]
Oct 30 13:08:43.671460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.671430    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"]
Oct 30 13:08:43.674939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:43.674910    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8c35df-57f5d4c84c-kddq6"]
Oct 30 13:08:44.077133 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.077099    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707e9065-0497-4d0a-a4b4-8241ea0546e2" path="/var/lib/kubelet/pods/707e9065-0497-4d0a-a4b4-8241ea0546e2/volumes"
Oct 30 13:08:44.077627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.077613    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e935a4c8-fd70-4e86-833f-a6d5babf2f91" path="/var/lib/kubelet/pods/e935a4c8-fd70-4e86-833f-a6d5babf2f91/volumes"
Oct 30 13:08:44.102047 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.102015    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"]
Oct 30 13:08:44.102333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.102275    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz" podUID="0917c4df-8abd-4e0f-837d-254197edfb36" containerName="maas-api" containerID="cri-o://4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073" gracePeriod=30
Oct 30 13:08:44.303832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.303801    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"]
Oct 30 13:08:44.304035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.303992    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk" podUID="22b2bb3b-68e9-4282-85bc-25b611741d46" containerName="payload-processing" containerID="cri-o://1ffa970692da2b6e431f7b3eeaf7cf3556c159570e5ee29517c31fa316f585f6" gracePeriod=30
Oct 30 13:08:44.347189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.347150    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"]
Oct 30 13:08:44.347438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.347399    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk" podUID="79a724fb-1295-4798-848b-07cdd09344e5" containerName="payload-pre-processing" containerID="cri-o://b860014add2b2f688ba910ba0e6ad7a5e6b6316dc0f74c2e3b7183cf963f4ed0" gracePeriod=30
Oct 30 13:08:44.481409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.481382    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:08:44.553367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.553336    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls\") pod \"0917c4df-8abd-4e0f-837d-254197edfb36\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") "
Oct 30 13:08:44.553747 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.553439    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0917c4df-8abd-4e0f-837d-254197edfb36-kube-api-access-rwnc5\" (UniqueName: \"kubernetes.io/projected/0917c4df-8abd-4e0f-837d-254197edfb36-kube-api-access-rwnc5\") pod \"0917c4df-8abd-4e0f-837d-254197edfb36\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") "
Oct 30 13:08:44.553888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.553868    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0917c4df-8abd-4e0f-837d-254197edfb36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0917c4df-8abd-4e0f-837d-254197edfb36-openshift-service-ca\") pod \"0917c4df-8abd-4e0f-837d-254197edfb36\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") "
Oct 30 13:08:44.554250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.554227    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0917c4df-8abd-4e0f-837d-254197edfb36-openshift-service-ca" pod "0917c4df-8abd-4e0f-837d-254197edfb36" (UID: "0917c4df-8abd-4e0f-837d-254197edfb36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:44.554447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.554430    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-metrics-certs\") pod \"0917c4df-8abd-4e0f-837d-254197edfb36\" (UID: \"0917c4df-8abd-4e0f-837d-254197edfb36\") "
Oct 30 13:08:44.555523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.555498    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0917c4df-8abd-4e0f-837d-254197edfb36-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:44.556386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.556360    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls" pod "0917c4df-8abd-4e0f-837d-254197edfb36" (UID: "0917c4df-8abd-4e0f-837d-254197edfb36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:44.557235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.557211    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0917c4df-8abd-4e0f-837d-254197edfb36-kube-api-access-rwnc5" pod "0917c4df-8abd-4e0f-837d-254197edfb36" (UID: "0917c4df-8abd-4e0f-837d-254197edfb36"). InnerVolumeSpecName "kube-api-access-rwnc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:44.557391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.557373    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-metrics-certs" pod "0917c4df-8abd-4e0f-837d-254197edfb36" (UID: "0917c4df-8abd-4e0f-837d-254197edfb36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:08:44.574320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.573345    4429 generic.go:356] "Generic (PLEG): container finished" podID="22b2bb3b-68e9-4282-85bc-25b611741d46" containerID="1ffa970692da2b6e431f7b3eeaf7cf3556c159570e5ee29517c31fa316f585f6" exitCode=0
Oct 30 13:08:44.574320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.573415    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk" event={"ID":"22b2bb3b-68e9-4282-85bc-25b611741d46","Type":"ContainerDied","Data":"1ffa970692da2b6e431f7b3eeaf7cf3556c159570e5ee29517c31fa316f585f6"}
Oct 30 13:08:44.575341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.575316    4429 generic.go:356] "Generic (PLEG): container finished" podID="79a724fb-1295-4798-848b-07cdd09344e5" containerID="b860014add2b2f688ba910ba0e6ad7a5e6b6316dc0f74c2e3b7183cf963f4ed0" exitCode=0
Oct 30 13:08:44.575463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.575405    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk" event={"ID":"79a724fb-1295-4798-848b-07cdd09344e5","Type":"ContainerDied","Data":"b860014add2b2f688ba910ba0e6ad7a5e6b6316dc0f74c2e3b7183cf963f4ed0"}
Oct 30 13:08:44.577937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.577913    4429 generic.go:356] "Generic (PLEG): container finished" podID="0917c4df-8abd-4e0f-837d-254197edfb36" containerID="4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073" exitCode=0
Oct 30 13:08:44.578051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.578005    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz" event={"ID":"0917c4df-8abd-4e0f-837d-254197edfb36","Type":"ContainerDied","Data":"4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073"}
Oct 30 13:08:44.578051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.578045    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz" event={"ID":"0917c4df-8abd-4e0f-837d-254197edfb36","Type":"ContainerDied","Data":"c4cee6cf8820a0cee505017d2f91f011e927fd1fe616c1608126d5b6185261ff"}
Oct 30 13:08:44.578151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.578073    4429 scope.go:122] "RemoveContainer" containerID="4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073"
Oct 30 13:08:44.578204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.578190    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"
Oct 30 13:08:44.591681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.591659    4429 scope.go:122] "RemoveContainer" containerID="4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073"
Oct 30 13:08:44.591971 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:08:44.591951    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073\": container with ID starting with 4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073 not found: ID does not exist" containerID="4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073"
Oct 30 13:08:44.592053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.591979    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073"} err="failed to get container status \"4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073\": rpc error: code = NotFound desc = could not find container \"4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073\": container with ID starting with 4f2b734bcb359ed7088c560103c8dade28401b86d61b92d920b89defe6723073 not found: ID does not exist"
Oct 30 13:08:44.644532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.644127    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"]
Oct 30 13:08:44.658253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.658038    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:44.658253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.658071    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0917c4df-8abd-4e0f-837d-254197edfb36-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:44.658253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.658086    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwnc5\" (UniqueName: \"kubernetes.io/projected/0917c4df-8abd-4e0f-837d-254197edfb36-kube-api-access-rwnc5\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:44.701329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.695577    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:08:44.752933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.752894    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6e0e74-5f9b49ffcf-hmlqz"]
Oct 30 13:08:44.803668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.801171    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:08:44.860812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.860777    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b2bb3b-68e9-4282-85bc-25b611741d46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b2bb3b-68e9-4282-85bc-25b611741d46-plugins-config-volume\") pod \"22b2bb3b-68e9-4282-85bc-25b611741d46\" (UID: \"22b2bb3b-68e9-4282-85bc-25b611741d46\") "
Oct 30 13:08:44.861019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.860902    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b2bb3b-68e9-4282-85bc-25b611741d46-kube-api-access-k68sd\" (UniqueName: \"kubernetes.io/projected/22b2bb3b-68e9-4282-85bc-25b611741d46-kube-api-access-k68sd\") pod \"22b2bb3b-68e9-4282-85bc-25b611741d46\" (UID: \"22b2bb3b-68e9-4282-85bc-25b611741d46\") "
Oct 30 13:08:44.861019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.860968    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a724fb-1295-4798-848b-07cdd09344e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a724fb-1295-4798-848b-07cdd09344e5-plugins-config-volume\") pod \"79a724fb-1295-4798-848b-07cdd09344e5\" (UID: \"79a724fb-1295-4798-848b-07cdd09344e5\") "
Oct 30 13:08:44.861117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.861097    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a724fb-1295-4798-848b-07cdd09344e5-kube-api-access-djmsq\" (UniqueName: \"kubernetes.io/projected/79a724fb-1295-4798-848b-07cdd09344e5-kube-api-access-djmsq\") pod \"79a724fb-1295-4798-848b-07cdd09344e5\" (UID: \"79a724fb-1295-4798-848b-07cdd09344e5\") "
Oct 30 13:08:44.861463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.861256    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b2bb3b-68e9-4282-85bc-25b611741d46-plugins-config-volume" pod "22b2bb3b-68e9-4282-85bc-25b611741d46" (UID: "22b2bb3b-68e9-4282-85bc-25b611741d46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:44.861611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.861499    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a724fb-1295-4798-848b-07cdd09344e5-plugins-config-volume" pod "79a724fb-1295-4798-848b-07cdd09344e5" (UID: "79a724fb-1295-4798-848b-07cdd09344e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:44.861812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.861792    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b2bb3b-68e9-4282-85bc-25b611741d46-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:44.861895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.861813    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a724fb-1295-4798-848b-07cdd09344e5-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:44.864064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.864041    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b2bb3b-68e9-4282-85bc-25b611741d46-kube-api-access-k68sd" pod "22b2bb3b-68e9-4282-85bc-25b611741d46" (UID: "22b2bb3b-68e9-4282-85bc-25b611741d46"). InnerVolumeSpecName "kube-api-access-k68sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:44.865051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.865031    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a724fb-1295-4798-848b-07cdd09344e5-kube-api-access-djmsq" pod "79a724fb-1295-4798-848b-07cdd09344e5" (UID: "79a724fb-1295-4798-848b-07cdd09344e5"). InnerVolumeSpecName "kube-api-access-djmsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:44.962895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.962853    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djmsq\" (UniqueName: \"kubernetes.io/projected/79a724fb-1295-4798-848b-07cdd09344e5-kube-api-access-djmsq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:44.962895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:44.962882    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k68sd\" (UniqueName: \"kubernetes.io/projected/22b2bb3b-68e9-4282-85bc-25b611741d46-kube-api-access-k68sd\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:45.585392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.585361    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"
Oct 30 13:08:45.585886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.585409    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk" event={"ID":"22b2bb3b-68e9-4282-85bc-25b611741d46","Type":"ContainerDied","Data":"c8f40a50badb0396357f6ddc73ea488417f39b756ffcdbbeb31f9feb7408f7c7"}
Oct 30 13:08:45.585886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.585477    4429 scope.go:122] "RemoveContainer" containerID="1ffa970692da2b6e431f7b3eeaf7cf3556c159570e5ee29517c31fa316f585f6"
Oct 30 13:08:45.587231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.587035    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"
Oct 30 13:08:45.587231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.587059    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk" event={"ID":"79a724fb-1295-4798-848b-07cdd09344e5","Type":"ContainerDied","Data":"6724f9a13f8d01958c0d6380e9e651d299f48b77869e6808f97716a41ff69c3a"}
Oct 30 13:08:45.596714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.596613    4429 scope.go:122] "RemoveContainer" containerID="b860014add2b2f688ba910ba0e6ad7a5e6b6316dc0f74c2e3b7183cf963f4ed0"
Oct 30 13:08:45.673280 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.673245    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"]
Oct 30 13:08:45.681762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.681735    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6e0e74-696f4c45d7-4hvjk"]
Oct 30 13:08:45.746881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.746853    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"]
Oct 30 13:08:45.750653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:45.750628    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6e0e74-749fd9cc5d-2fwdk"]
Oct 30 13:08:46.075979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:46.075949    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0917c4df-8abd-4e0f-837d-254197edfb36" path="/var/lib/kubelet/pods/0917c4df-8abd-4e0f-837d-254197edfb36/volumes"
Oct 30 13:08:46.076387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:46.076369    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b2bb3b-68e9-4282-85bc-25b611741d46" path="/var/lib/kubelet/pods/22b2bb3b-68e9-4282-85bc-25b611741d46/volumes"
Oct 30 13:08:46.076764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:46.076748    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a724fb-1295-4798-848b-07cdd09344e5" path="/var/lib/kubelet/pods/79a724fb-1295-4798-848b-07cdd09344e5/volumes"
Oct 30 13:08:47.671484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.671451    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"]
Oct 30 13:08:47.672083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672070    4429 cpu_manager.go:414] "containerMap: removing container" podUID="707e9065-0497-4d0a-a4b4-8241ea0546e2" containerName="payload-pre-processing"
Oct 30 13:08:47.672140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672085    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707e9065-0497-4d0a-a4b4-8241ea0546e2" containerName="payload-pre-processing"
Oct 30 13:08:47.672140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672099    4429 cpu_manager.go:414] "containerMap: removing container" podUID="79a724fb-1295-4798-848b-07cdd09344e5" containerName="payload-pre-processing"
Oct 30 13:08:47.672140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672104    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a724fb-1295-4798-848b-07cdd09344e5" containerName="payload-pre-processing"
Oct 30 13:08:47.672140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672116    4429 cpu_manager.go:414] "containerMap: removing container" podUID="22b2bb3b-68e9-4282-85bc-25b611741d46" containerName="payload-processing"
Oct 30 13:08:47.672140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672120    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b2bb3b-68e9-4282-85bc-25b611741d46" containerName="payload-processing"
Oct 30 13:08:47.672140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672137    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0917c4df-8abd-4e0f-837d-254197edfb36" containerName="maas-api"
Oct 30 13:08:47.672356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672143    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0917c4df-8abd-4e0f-837d-254197edfb36" containerName="maas-api"
Oct 30 13:08:47.672356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672171    4429 cpu_manager.go:414] "containerMap: removing container" podUID="e935a4c8-fd70-4e86-833f-a6d5babf2f91" containerName="payload-processing"
Oct 30 13:08:47.672356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672176    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e935a4c8-fd70-4e86-833f-a6d5babf2f91" containerName="payload-processing"
Oct 30 13:08:47.672356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672255    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="e935a4c8-fd70-4e86-833f-a6d5babf2f91" containerName="payload-processing"
Oct 30 13:08:47.672356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672264    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="707e9065-0497-4d0a-a4b4-8241ea0546e2" containerName="payload-pre-processing"
Oct 30 13:08:47.672356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672270    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a724fb-1295-4798-848b-07cdd09344e5" containerName="payload-pre-processing"
Oct 30 13:08:47.672356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672276    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b2bb3b-68e9-4282-85bc-25b611741d46" containerName="payload-processing"
Oct 30 13:08:47.672356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.672321    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0917c4df-8abd-4e0f-837d-254197edfb36" containerName="maas-api"
Oct 30 13:08:47.676765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.676743    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:47.697392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.697364    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a66ac8b-b9bd-4335-bb8e-91d102a88280-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj\" (UID: \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:47.697392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.697382    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d84f47\""
Oct 30 13:08:47.697548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.697466    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmsg\" (UniqueName: \"kubernetes.io/projected/1a66ac8b-b9bd-4335-bb8e-91d102a88280-kube-api-access-mcmsg\") pod \"payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj\" (UID: \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:47.697548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.697514    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d84f47-dockercfg-jnnl7\""
Oct 30 13:08:47.717006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.716973    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"]
Oct 30 13:08:47.799225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.799193    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a66ac8b-b9bd-4335-bb8e-91d102a88280-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj\" (UID: \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:47.799459 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.799307    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmsg\" (UniqueName: \"kubernetes.io/projected/1a66ac8b-b9bd-4335-bb8e-91d102a88280-kube-api-access-mcmsg\") pod \"payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj\" (UID: \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:47.800343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.800218    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a66ac8b-b9bd-4335-bb8e-91d102a88280-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj\" (UID: \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:47.829068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.829034    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmsg\" (UniqueName: \"kubernetes.io/projected/1a66ac8b-b9bd-4335-bb8e-91d102a88280-kube-api-access-mcmsg\") pod \"payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj\" (UID: \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:47.989853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:47.989822    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:48.313326 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:08:48.300840    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a66ac8b_b9bd_4335_bb8e_91d102a88280.slice/crio-40d01b47b84e219c0028b4af23392391b5a7da1ed577a30b84119eebce92e9ed WatchSource:0}: Error finding container 40d01b47b84e219c0028b4af23392391b5a7da1ed577a30b84119eebce92e9ed: Status 404 returned error can't find the container with id 40d01b47b84e219c0028b4af23392391b5a7da1ed577a30b84119eebce92e9ed
Oct 30 13:08:48.313326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:48.307152    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"]
Oct 30 13:08:48.616069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:48.615996    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj" event={"ID":"1a66ac8b-b9bd-4335-bb8e-91d102a88280","Type":"ContainerStarted","Data":"150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3"}
Oct 30 13:08:48.616069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:48.616047    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj" event={"ID":"1a66ac8b-b9bd-4335-bb8e-91d102a88280","Type":"ContainerStarted","Data":"40d01b47b84e219c0028b4af23392391b5a7da1ed577a30b84119eebce92e9ed"}
Oct 30 13:08:48.616237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:48.616105    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:08:49.898231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:49.898178    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj" podStartSLOduration=2.898162026 podStartE2EDuration="2.898162026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:48.665622704 +0000 UTC m=+2186.721116529" watchObservedRunningTime="2026-10-30 13:08:49.898162026 +0000 UTC m=+2187.953655896"
Oct 30 13:08:49.899376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:49.899352    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"]
Oct 30 13:08:49.899603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:49.899582    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy" containerID="cri-o://6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb" gracePeriod=30
Oct 30 13:08:55.180177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.180095    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:08:55.306228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306195    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-socket\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.306415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306267    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-certs\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.306415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306323    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istiod-ca-cert\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.306503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306418    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-podinfo\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.306503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306468    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-credential-socket\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.306608 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306505    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-data\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.306608 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306553    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-socket" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:55.306715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306594    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-certs" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:55.306715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306618    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-kube-api-access-l7qjt\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-kube-api-access-l7qjt\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.306715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306670    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-envoy\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.306931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306773    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-credential-socket" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:55.308058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306815    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-token\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-token\") pod \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\" (UID: \"5058c878-b244-4c68-8b2d-b5f7dc0c7391\") "
Oct 30 13:08:55.308058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306856    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-data" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:55.308058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.306965    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istiod-ca-cert" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:08:55.308058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.307942    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.308058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.307967    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.308058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.307982    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.308058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.307996    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.308058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.308009    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.309522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.309491    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-envoy" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:08:55.309614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.309542    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-kube-api-access-l7qjt" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "kube-api-access-l7qjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:55.309614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.309575    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-token" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:55.309698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.309636    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-podinfo" pod "5058c878-b244-4c68-8b2d-b5f7dc0c7391" (UID: "5058c878-b244-4c68-8b2d-b5f7dc0c7391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:08:55.416628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.416517    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.416628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.416553    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.416628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.416569    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qjt\" (UniqueName: \"kubernetes.io/projected/5058c878-b244-4c68-8b2d-b5f7dc0c7391-kube-api-access-l7qjt\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.416628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.416586    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5058c878-b244-4c68-8b2d-b5f7dc0c7391-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:08:55.669039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.669006    4429 generic.go:356] "Generic (PLEG): container finished" podID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerID="6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb" exitCode=0
Oct 30 13:08:55.669211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.669051    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" event={"ID":"5058c878-b244-4c68-8b2d-b5f7dc0c7391","Type":"ContainerDied","Data":"6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb"}
Oct 30 13:08:55.669211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.669123    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm" event={"ID":"5058c878-b244-4c68-8b2d-b5f7dc0c7391","Type":"ContainerDied","Data":"59eabfc9710b682e234f2674fb5cc5032c1942dc0c57ca1152001c7e18cc9d35"}
Oct 30 13:08:55.669211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.669064    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"
Oct 30 13:08:55.669211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.669152    4429 scope.go:122] "RemoveContainer" containerID="6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb"
Oct 30 13:08:55.680755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.680728    4429 scope.go:122] "RemoveContainer" containerID="eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026"
Oct 30 13:08:55.690267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.690246    4429 scope.go:122] "RemoveContainer" containerID="6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb"
Oct 30 13:08:55.690592 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:08:55.690572    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb\": container with ID starting with 6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb not found: ID does not exist" containerID="6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb"
Oct 30 13:08:55.690696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.690598    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb"} err="failed to get container status \"6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb\": rpc error: code = NotFound desc = could not find container \"6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb\": container with ID starting with 6745888183ed9df705915f7d3827656960e213888248ba0b39bf05f3adce60fb not found: ID does not exist"
Oct 30 13:08:55.690696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.690615    4429 scope.go:122] "RemoveContainer" containerID="eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026"
Oct 30 13:08:55.690854 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:08:55.690836    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026\": container with ID starting with eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026 not found: ID does not exist" containerID="eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026"
Oct 30 13:08:55.690889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.690860    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026"} err="failed to get container status \"eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026\": rpc error: code = NotFound desc = could not find container \"eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026\": container with ID starting with eeef37fd1aed4a69666540e84bd17b65ac07c9cd779158f2f9823d22f2633026 not found: ID does not exist"
Oct 30 13:08:55.730198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.730169    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"]
Oct 30 13:08:55.743754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:55.743730    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8c35df-openshift-default-6fb49dfb7c-z2tlm"]
Oct 30 13:08:56.076214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:56.076184    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" path="/var/lib/kubelet/pods/5058c878-b244-4c68-8b2d-b5f7dc0c7391/volumes"
Oct 30 13:08:59.625436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:08:59.625410    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:09:12.377010 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:09:12.376925    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:09:21.656622 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:21.656588    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"]
Oct 30 13:09:21.692663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:21.692635    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"]
Oct 30 13:09:23.649093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:23.649063    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"]
Oct 30 13:09:23.649624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:23.649252    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy" containerID="cri-o://ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07" gracePeriod=30
Oct 30 13:09:26.657194 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.657157    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh" podUID="8ca41b7d-1f5c-4a2e-b892-073a54c4547c" containerName="payload-processing" containerID="cri-o://c08a9f922b5f8ffbee8fc328818cd8e568f9ec82fddd44e9dc9293a7abe4c1d1" gracePeriod=25
Oct 30 13:09:26.693835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.693778    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d" podUID="3416cc31-7540-4662-9236-d05f9b7e2510" containerName="payload-pre-processing" containerID="cri-o://14a79d39534f411f83cc14ed74885dc278a3d60e4343c758f0f900ebc389d602" gracePeriod=25
Oct 30 13:09:26.823729 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:09:26.823702    4429 logging.go:55] [core] [Channel #2037 SubChannel #2038]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.118:50052", ServerName: "10.133.0.118:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.118:50052: connect: connection refused"
Oct 30 13:09:26.823729 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:09:26.823716    4429 logging.go:55] [core] [Channel #2036 SubChannel #2039]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.117:50052", ServerName: "10.133.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.117:50052: connect: connection refused"
Oct 30 13:09:26.901944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.901907    4429 generic.go:356] "Generic (PLEG): container finished" podID="8ca41b7d-1f5c-4a2e-b892-073a54c4547c" containerID="c08a9f922b5f8ffbee8fc328818cd8e568f9ec82fddd44e9dc9293a7abe4c1d1" exitCode=0
Oct 30 13:09:26.902252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.902139    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh" event={"ID":"8ca41b7d-1f5c-4a2e-b892-073a54c4547c","Type":"ContainerDied","Data":"c08a9f922b5f8ffbee8fc328818cd8e568f9ec82fddd44e9dc9293a7abe4c1d1"}
Oct 30 13:09:26.904691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.904664    4429 generic.go:356] "Generic (PLEG): container finished" podID="3416cc31-7540-4662-9236-d05f9b7e2510" containerID="14a79d39534f411f83cc14ed74885dc278a3d60e4343c758f0f900ebc389d602" exitCode=0
Oct 30 13:09:26.904874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.904724    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d" event={"ID":"3416cc31-7540-4662-9236-d05f9b7e2510","Type":"ContainerDied","Data":"14a79d39534f411f83cc14ed74885dc278a3d60e4343c758f0f900ebc389d602"}
Oct 30 13:09:26.984306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.984253    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:09:26.992486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.992453    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-plugins-config-volume\") pod \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\" (UID: \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\") "
Oct 30 13:09:26.992668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.992648    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-kube-api-access-clbxh\" (UniqueName: \"kubernetes.io/projected/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-kube-api-access-clbxh\") pod \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\" (UID: \"8ca41b7d-1f5c-4a2e-b892-073a54c4547c\") "
Oct 30 13:09:26.993153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.993127    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-plugins-config-volume" pod "8ca41b7d-1f5c-4a2e-b892-073a54c4547c" (UID: "8ca41b7d-1f5c-4a2e-b892-073a54c4547c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:26.995908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:26.995875    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-kube-api-access-clbxh" pod "8ca41b7d-1f5c-4a2e-b892-073a54c4547c" (UID: "8ca41b7d-1f5c-4a2e-b892-073a54c4547c"). InnerVolumeSpecName "kube-api-access-clbxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:27.032815 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.032795    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:09:27.093822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.093791    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3416cc31-7540-4662-9236-d05f9b7e2510-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3416cc31-7540-4662-9236-d05f9b7e2510-plugins-config-volume\") pod \"3416cc31-7540-4662-9236-d05f9b7e2510\" (UID: \"3416cc31-7540-4662-9236-d05f9b7e2510\") "
Oct 30 13:09:27.094047 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.093849    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3416cc31-7540-4662-9236-d05f9b7e2510-kube-api-access-qbrnd\" (UniqueName: \"kubernetes.io/projected/3416cc31-7540-4662-9236-d05f9b7e2510-kube-api-access-qbrnd\") pod \"3416cc31-7540-4662-9236-d05f9b7e2510\" (UID: \"3416cc31-7540-4662-9236-d05f9b7e2510\") "
Oct 30 13:09:27.094233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.094203    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3416cc31-7540-4662-9236-d05f9b7e2510-plugins-config-volume" pod "3416cc31-7540-4662-9236-d05f9b7e2510" (UID: "3416cc31-7540-4662-9236-d05f9b7e2510"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:27.094421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.094400    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clbxh\" (UniqueName: \"kubernetes.io/projected/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-kube-api-access-clbxh\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:27.094501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.094428    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3416cc31-7540-4662-9236-d05f9b7e2510-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:27.094501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.094446    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca41b7d-1f5c-4a2e-b892-073a54c4547c-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:27.096646 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.096623    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3416cc31-7540-4662-9236-d05f9b7e2510-kube-api-access-qbrnd" pod "3416cc31-7540-4662-9236-d05f9b7e2510" (UID: "3416cc31-7540-4662-9236-d05f9b7e2510"). InnerVolumeSpecName "kube-api-access-qbrnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:27.195806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.195778    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrnd\" (UniqueName: \"kubernetes.io/projected/3416cc31-7540-4662-9236-d05f9b7e2510-kube-api-access-qbrnd\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:27.654712 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.654679    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"]
Oct 30 13:09:27.654937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.654914    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc" podUID="f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" containerName="maas-api" containerID="cri-o://c95651700e57c63588fa3be21bfebc03a649f32a932d535266d474330d3ba56c" gracePeriod=30
Oct 30 13:09:27.824064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.824006    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh" podUID="8ca41b7d-1f5c-4a2e-b892-073a54c4547c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.118:50052\" within 1s: context deadline exceeded"
Oct 30 13:09:27.824513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.824103    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d" podUID="3416cc31-7540-4662-9236-d05f9b7e2510" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.117:50052\" within 1s: context deadline exceeded"
Oct 30 13:09:27.824513 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:09:27.824180    4429 logging.go:55] [core] [Channel #2036 SubChannel #2039]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.117:50052", ServerName: "10.133.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.117:50052: operation was canceled"
Oct 30 13:09:27.965658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.963651    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d" event={"ID":"3416cc31-7540-4662-9236-d05f9b7e2510","Type":"ContainerDied","Data":"e3bb37f2d6d9b53b6fc00baded018b5dd82f431ca4a35a8dd10c0cc1aff6bb95"}
Oct 30 13:09:27.965658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.963738    4429 scope.go:122] "RemoveContainer" containerID="14a79d39534f411f83cc14ed74885dc278a3d60e4343c758f0f900ebc389d602"
Oct 30 13:09:27.965658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.963933    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"
Oct 30 13:09:27.977574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.977534    4429 generic.go:356] "Generic (PLEG): container finished" podID="f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" containerID="c95651700e57c63588fa3be21bfebc03a649f32a932d535266d474330d3ba56c" exitCode=0
Oct 30 13:09:27.977713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.977597    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc" event={"ID":"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b","Type":"ContainerDied","Data":"c95651700e57c63588fa3be21bfebc03a649f32a932d535266d474330d3ba56c"}
Oct 30 13:09:27.979534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.979510    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh" event={"ID":"8ca41b7d-1f5c-4a2e-b892-073a54c4547c","Type":"ContainerDied","Data":"2f98fd4611f432998a26f8ca28cad0eeaeb93c994d90379dfef15097d7ee5c39"}
Oct 30 13:09:27.979624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.979593    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"
Oct 30 13:09:27.988615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.988594    4429 scope.go:122] "RemoveContainer" containerID="c08a9f922b5f8ffbee8fc328818cd8e568f9ec82fddd44e9dc9293a7abe4c1d1"
Oct 30 13:09:27.995054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:27.995026    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:09:28.012662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.012634    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-openshift-service-ca\") pod \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") "
Oct 30 13:09:28.012845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.012760    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-maas-api-tls\") pod \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") "
Oct 30 13:09:28.012845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.012820    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-metrics-certs\") pod \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") "
Oct 30 13:09:28.012985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.012882    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-kube-api-access-pctkd\" (UniqueName: \"kubernetes.io/projected/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-kube-api-access-pctkd\") pod \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\" (UID: \"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b\") "
Oct 30 13:09:28.013071 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.013042    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-openshift-service-ca" pod "f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" (UID: "f19bb56a-2a1b-4f09-9ac2-fd7354b1824b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:28.013507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.013485    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:28.015034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.015008    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-maas-api-tls" pod "f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" (UID: "f19bb56a-2a1b-4f09-9ac2-fd7354b1824b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:28.015687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.015658    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-kube-api-access-pctkd" pod "f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" (UID: "f19bb56a-2a1b-4f09-9ac2-fd7354b1824b"). InnerVolumeSpecName "kube-api-access-pctkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:28.015687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.015670    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-metrics-certs" pod "f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" (UID: "f19bb56a-2a1b-4f09-9ac2-fd7354b1824b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:09:28.048876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.048837    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"]
Oct 30 13:09:28.063976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.063723    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e954cd-7fddbb56f9-bpv9d"]
Oct 30 13:09:28.088318 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.081963    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3416cc31-7540-4662-9236-d05f9b7e2510" path="/var/lib/kubelet/pods/3416cc31-7540-4662-9236-d05f9b7e2510/volumes"
Oct 30 13:09:28.124777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.118494    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:28.124777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.118518    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pctkd\" (UniqueName: \"kubernetes.io/projected/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-kube-api-access-pctkd\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:28.124777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.118532    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:28.158066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.155964    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"]
Oct 30 13:09:28.164628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.162923    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e954cd-699b946766-plqzh"]
Oct 30 13:09:28.737278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.737256    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2_2a1b7e09-67f1-4df1-b385-ae9ae8370634/payload-processing/0.log"
Oct 30 13:09:28.931705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.931625    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 13:09:28.989100 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.989072    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"
Oct 30 13:09:28.989367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.989071    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc" event={"ID":"f19bb56a-2a1b-4f09-9ac2-fd7354b1824b","Type":"ContainerDied","Data":"6857e776e4399eb795a1798f50404a02d86e8d244d16ba8cb0c1a846a1be80c4"}
Oct 30 13:09:28.989367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.989212    4429 scope.go:122] "RemoveContainer" containerID="c95651700e57c63588fa3be21bfebc03a649f32a932d535266d474330d3ba56c"
Oct 30 13:09:28.992023 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.991960    4429 generic.go:356] "Generic (PLEG): container finished" podID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerID="ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07" exitCode=0
Oct 30 13:09:28.992877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.992045    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" event={"ID":"11715a29-3dfa-44d8-8eaa-61ecc1f56143","Type":"ContainerDied","Data":"ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07"}
Oct 30 13:09:28.992877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.992092    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt" event={"ID":"11715a29-3dfa-44d8-8eaa-61ecc1f56143","Type":"ContainerDied","Data":"e027862355941cfd16906f55339796a980755606c6dbede9e2d969b7881f9bf0"}
Oct 30 13:09:28.992877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:28.992177    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"
Oct 30 13:09:29.004923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.004877    4429 scope.go:122] "RemoveContainer" containerID="ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07"
Oct 30 13:09:29.017375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.017351    4429 scope.go:122] "RemoveContainer" containerID="ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07"
Oct 30 13:09:29.017775 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:09:29.017752    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07\": container with ID starting with ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07 not found: ID does not exist" containerID="ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07"
Oct 30 13:09:29.017865 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.017788    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07"} err="failed to get container status \"ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07\": rpc error: code = NotFound desc = could not find container \"ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07\": container with ID starting with ebdc89bd4ff4a5d4c9de235e8b90f8363f2a9dce5942c66626f2b9d88efe9a07 not found: ID does not exist"
Oct 30 13:09:29.030662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.030640    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-podinfo\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.030791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.030681    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-socket\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.030791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.030718    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-token\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-token\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.030920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.030794    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istiod-ca-cert\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.031001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.030973    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-socket" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:29.031181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031160    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-envoy\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.031244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031184    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istiod-ca-cert" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:29.031327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031312    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-credential-socket\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.031403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031357    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-data\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.031403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031396    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-certs\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.031512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031459    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-kube-api-access-27tht\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-kube-api-access-27tht\") pod \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\" (UID: \"11715a29-3dfa-44d8-8eaa-61ecc1f56143\") "
Oct 30 13:09:29.031621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031603    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-credential-socket" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:29.031677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031638    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-data" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:29.031740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.031722    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-certs" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:29.032103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.032070    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.032103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.032095    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.032246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.032110    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.032246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.032121    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.032246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.032134    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.033562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.033539    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-token" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:29.033702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.033673    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-podinfo" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:29.033983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.033960    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-envoy" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:29.034167 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.034146    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-kube-api-access-27tht" pod "11715a29-3dfa-44d8-8eaa-61ecc1f56143" (UID: "11715a29-3dfa-44d8-8eaa-61ecc1f56143"). InnerVolumeSpecName "kube-api-access-27tht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:29.066704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.066674    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"]
Oct 30 13:09:29.080121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.080096    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e954cd-d9b945bf4-fv8xc"]
Oct 30 13:09:29.134169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.134087    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.134344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.134209    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.134344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.134261    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11715a29-3dfa-44d8-8eaa-61ecc1f56143-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.134344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.134278    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27tht\" (UniqueName: \"kubernetes.io/projected/11715a29-3dfa-44d8-8eaa-61ecc1f56143-kube-api-access-27tht\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:29.356853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.356824    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"]
Oct 30 13:09:29.373997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:29.373963    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-tchxt"]
Oct 30 13:09:30.076745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:30.076713    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" path="/var/lib/kubelet/pods/11715a29-3dfa-44d8-8eaa-61ecc1f56143/volumes"
Oct 30 13:09:30.077126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:30.077073    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca41b7d-1f5c-4a2e-b892-073a54c4547c" path="/var/lib/kubelet/pods/8ca41b7d-1f5c-4a2e-b892-073a54c4547c/volumes"
Oct 30 13:09:30.077475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:30.077460    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" path="/var/lib/kubelet/pods/f19bb56a-2a1b-4f09-9ac2-fd7354b1824b/volumes"
Oct 30 13:09:36.588443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:36.588403    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"]
Oct 30 13:09:37.055638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:37.055601    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy" containerID="cri-o://17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4" gracePeriod=30
Oct 30 13:09:39.079547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:39.079509    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:09:39.079961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:39.079568    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:09:42.324733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.324668    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:09:42.327674 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327651    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-socket\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.327773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327709    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c548aa-4723-4dec-8d9c-15c127040139-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c548aa-4723-4dec-8d9c-15c127040139-istiod-ca-cert\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.327773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327754    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-istio-token\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-istio-token\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.327851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327796    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-credential-socket\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.327851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327827    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-kube-api-access-9r87s\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-kube-api-access-9r87s\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.327947 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327864    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64c548aa-4723-4dec-8d9c-15c127040139-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c548aa-4723-4dec-8d9c-15c127040139-istio-podinfo\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.327947 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327894    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-certs\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.327947 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327920    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-socket" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:42.328104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.327965    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-data\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.328104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328045    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-envoy\") pod \"64c548aa-4723-4dec-8d9c-15c127040139\" (UID: \"64c548aa-4723-4dec-8d9c-15c127040139\") "
Oct 30 13:09:42.328228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328208    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-credential-socket" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:42.328306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328224    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c548aa-4723-4dec-8d9c-15c127040139-istiod-ca-cert" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:42.328306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328247    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-data" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:42.328433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328363    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-certs" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:42.328729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328658    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.328729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328687    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c548aa-4723-4dec-8d9c-15c127040139-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.328729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328696    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.328729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328705    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.328729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.328713    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.330177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.330151    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-istio-token" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:42.330177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.330169    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-kube-api-access-9r87s" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "kube-api-access-9r87s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:42.330725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.330708    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-envoy" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:42.330804 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.330721    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64c548aa-4723-4dec-8d9c-15c127040139-istio-podinfo" pod "64c548aa-4723-4dec-8d9c-15c127040139" (UID: "64c548aa-4723-4dec-8d9c-15c127040139"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:42.372615 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:09:42.372575    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:09:42.429940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.429899    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.430121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.429946    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r87s\" (UniqueName: \"kubernetes.io/projected/64c548aa-4723-4dec-8d9c-15c127040139-kube-api-access-9r87s\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.430121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.429964    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c548aa-4723-4dec-8d9c-15c127040139-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:42.430121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:42.429981    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c548aa-4723-4dec-8d9c-15c127040139-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:43.096542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.096504    4429 generic.go:356] "Generic (PLEG): container finished" podID="64c548aa-4723-4dec-8d9c-15c127040139" containerID="17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4" exitCode=0
Oct 30 13:09:43.096741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.096569    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" event={"ID":"64c548aa-4723-4dec-8d9c-15c127040139","Type":"ContainerDied","Data":"17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4"}
Oct 30 13:09:43.096741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.096596    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"
Oct 30 13:09:43.096741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.096624    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh" event={"ID":"64c548aa-4723-4dec-8d9c-15c127040139","Type":"ContainerDied","Data":"ae8df21041e4bb1f667c86539af1ebd34834a3da238e049efc3efab769fb0acf"}
Oct 30 13:09:43.096741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.096647    4429 scope.go:122] "RemoveContainer" containerID="17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4"
Oct 30 13:09:43.107912 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.107599    4429 scope.go:122] "RemoveContainer" containerID="002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95"
Oct 30 13:09:43.121387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.121356    4429 scope.go:122] "RemoveContainer" containerID="17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4"
Oct 30 13:09:43.121766 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:09:43.121743    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4\": container with ID starting with 17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4 not found: ID does not exist" containerID="17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4"
Oct 30 13:09:43.121822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.121777    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4"} err="failed to get container status \"17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4\": rpc error: code = NotFound desc = could not find container \"17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4\": container with ID starting with 17387ba82fc828e82dd339d5479539def7b26d1a556e01c179c69124835dc3e4 not found: ID does not exist"
Oct 30 13:09:43.121822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.121797    4429 scope.go:122] "RemoveContainer" containerID="002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95"
Oct 30 13:09:43.122077 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:09:43.122057    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95\": container with ID starting with 002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95 not found: ID does not exist" containerID="002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95"
Oct 30 13:09:43.122139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.122083    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95"} err="failed to get container status \"002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95\": rpc error: code = NotFound desc = could not find container \"002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95\": container with ID starting with 002c6b8f39c79d11ffb07a327fbd1874c13c9e462bd2523a3c1ed02e6c4a4c95 not found: ID does not exist"
Oct 30 13:09:43.172090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.172049    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"]
Oct 30 13:09:43.186038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:43.186002    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e954cd-openshift-default-7557cd58fc-dqhvh"]
Oct 30 13:09:44.075826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:44.075794    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c548aa-4723-4dec-8d9c-15c127040139" path="/var/lib/kubelet/pods/64c548aa-4723-4dec-8d9c-15c127040139/volumes"
Oct 30 13:09:46.510074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.510038    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"]
Oct 30 13:09:46.511874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511835    4429 cpu_manager.go:414] "containerMap: removing container" podUID="3416cc31-7540-4662-9236-d05f9b7e2510" containerName="payload-pre-processing"
Oct 30 13:09:46.511874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511863    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3416cc31-7540-4662-9236-d05f9b7e2510" containerName="payload-pre-processing"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511913    4429 cpu_manager.go:414] "containerMap: removing container" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511921    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511934    4429 cpu_manager.go:414] "containerMap: removing container" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511942    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511951    4429 cpu_manager.go:414] "containerMap: removing container" podUID="f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" containerName="maas-api"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511957    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" containerName="maas-api"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511976    4429 cpu_manager.go:414] "containerMap: removing container" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511985    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.511995    4429 cpu_manager.go:414] "containerMap: removing container" podUID="8ca41b7d-1f5c-4a2e-b892-073a54c4547c" containerName="payload-processing"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512001    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca41b7d-1f5c-4a2e-b892-073a54c4547c" containerName="payload-processing"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512010    4429 cpu_manager.go:414] "containerMap: removing container" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy"
Oct 30 13:09:46.512088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512017    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy"
Oct 30 13:09:46.512726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512161    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy"
Oct 30 13:09:46.512726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512177    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="11715a29-3dfa-44d8-8eaa-61ecc1f56143" containerName="istio-proxy"
Oct 30 13:09:46.512726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512187    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19bb56a-2a1b-4f09-9ac2-fd7354b1824b" containerName="maas-api"
Oct 30 13:09:46.512726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512197    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy"
Oct 30 13:09:46.512726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512207    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy"
Oct 30 13:09:46.512726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512216    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca41b7d-1f5c-4a2e-b892-073a54c4547c" containerName="payload-processing"
Oct 30 13:09:46.512726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512224    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c548aa-4723-4dec-8d9c-15c127040139" containerName="istio-proxy"
Oct 30 13:09:46.512726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.512239    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="3416cc31-7540-4662-9236-d05f9b7e2510" containerName="payload-pre-processing"
Oct 30 13:09:46.521326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.521251    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.541671 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.541641    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"]
Oct 30 13:09:46.551260 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.551227    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ffbba3-openshift-default-dockercfg-b7522\""
Oct 30 13:09:46.595405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.595366    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z594h\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-kube-api-access-z594h\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.595405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.595410    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-envoy\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.595628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.595431    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-credential-socket\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.595669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.595622    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-token\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.595785 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.595762    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-podinfo\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.595921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.595822    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-certs\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.595967 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.595944    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-socket\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.596023 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.595978    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-data\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.596069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.596041    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istiod-ca-cert\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.697482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697420    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z594h\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-kube-api-access-z594h\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.697707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697496    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-envoy\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.697707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697530    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-credential-socket\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.697707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697618    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-token\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.697892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697737    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-podinfo\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.697892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697790    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-certs\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.697892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697842    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-socket\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.697892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697872    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-data\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.698113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.697901    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istiod-ca-cert\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.699672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.699638    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-credential-socket\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.700410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.700199    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istiod-ca-cert\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.700624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.700569    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-data\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.700759 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.700734    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-certs\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.701102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.701079    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-socket\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.734884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.732727    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-envoy\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.746325 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.736399    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-podinfo\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.760334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.755955    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-token\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.760334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.758064    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z594h\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-kube-api-access-z594h\") pod \"e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") " pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:46.839909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:46.839875    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:47.178046 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:09:47.178007    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a805ee4_c7f9_4e63_a427_ec946740e9bf.slice/crio-503d2f7aa5b881e881d9c5946ae17db31f2070e5da9a896a600d2e0126a14522 WatchSource:0}: Error finding container 503d2f7aa5b881e881d9c5946ae17db31f2070e5da9a896a600d2e0126a14522: Status 404 returned error can't find the container with id 503d2f7aa5b881e881d9c5946ae17db31f2070e5da9a896a600d2e0126a14522
Oct 30 13:09:47.178197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:47.178088    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"]
Oct 30 13:09:47.180976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:47.180936    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:47.181096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:47.181012    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:47.181096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:47.181045    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:48.132379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:48.132338    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" event={"ID":"1a805ee4-c7f9-4e63-a427-ec946740e9bf","Type":"ContainerStarted","Data":"c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794"}
Oct 30 13:09:48.132866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:48.132398    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" event={"ID":"1a805ee4-c7f9-4e63-a427-ec946740e9bf","Type":"ContainerStarted","Data":"503d2f7aa5b881e881d9c5946ae17db31f2070e5da9a896a600d2e0126a14522"}
Oct 30 13:09:48.215807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:48.215746    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" podStartSLOduration=2.215729763 podStartE2EDuration="2.215729763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:48.199045113 +0000 UTC m=+2246.254538936" watchObservedRunningTime="2026-10-30 13:09:48.215729763 +0000 UTC m=+2246.271223586"
Oct 30 13:09:48.699709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:48.699676    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"]
Oct 30 13:09:48.733845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:48.733807    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"]
Oct 30 13:09:48.840857 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:48.840815    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:48.847873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:48.847841    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:49.138917 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:49.138831    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:49.140067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:49.140050    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:09:52.839809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.839192    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"]
Oct 30 13:09:52.840499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.840320    4429 cpu_manager.go:414] "containerMap: removing container" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy"
Oct 30 13:09:52.840499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.840342    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5058c878-b244-4c68-8b2d-b5f7dc0c7391" containerName="istio-proxy"
Oct 30 13:09:52.904312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.892905    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"]
Oct 30 13:09:52.904312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.896334    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:52.908317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.904782    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"]
Oct 30 13:09:52.926208 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.924301    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ffbba3\""
Oct 30 13:09:52.926208 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.925099    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ffbba3-dockercfg-lp9f8\""
Oct 30 13:09:52.959612 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:52.959585    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:53.001589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.001525    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"]
Oct 30 13:09:53.003760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.003728    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaeba70d-8acf-4150-83e5-9f95d6832a1a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq\" (UID: \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:53.004020 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.004000    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84n9s\" (UniqueName: \"kubernetes.io/projected/beaf5b45-651c-4ccc-89a2-70a03290b1d7-kube-api-access-84n9s\") pod \"payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58\" (UID: \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:53.004174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.004158    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5nc\" (UniqueName: \"kubernetes.io/projected/aaeba70d-8acf-4150-83e5-9f95d6832a1a-kube-api-access-zr5nc\") pod \"payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq\" (UID: \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:53.004543 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.004519    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beaf5b45-651c-4ccc-89a2-70a03290b1d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58\" (UID: \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:53.105807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.105710    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5nc\" (UniqueName: \"kubernetes.io/projected/aaeba70d-8acf-4150-83e5-9f95d6832a1a-kube-api-access-zr5nc\") pod \"payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq\" (UID: \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:53.105976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.105889    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beaf5b45-651c-4ccc-89a2-70a03290b1d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58\" (UID: \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:53.106019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.105982    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaeba70d-8acf-4150-83e5-9f95d6832a1a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq\" (UID: \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:53.106089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.106074    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84n9s\" (UniqueName: \"kubernetes.io/projected/beaf5b45-651c-4ccc-89a2-70a03290b1d7-kube-api-access-84n9s\") pod \"payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58\" (UID: \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:53.106728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.106704    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaeba70d-8acf-4150-83e5-9f95d6832a1a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq\" (UID: \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:53.106833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.106713    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beaf5b45-651c-4ccc-89a2-70a03290b1d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58\" (UID: \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:53.131631 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.131598    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84n9s\" (UniqueName: \"kubernetes.io/projected/beaf5b45-651c-4ccc-89a2-70a03290b1d7-kube-api-access-84n9s\") pod \"payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58\" (UID: \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:53.131807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.131746    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5nc\" (UniqueName: \"kubernetes.io/projected/aaeba70d-8acf-4150-83e5-9f95d6832a1a-kube-api-access-zr5nc\") pod \"payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq\" (UID: \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:53.263564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.263518    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:53.345773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.345740    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:53.578159 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:09:53.578122    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeaf5b45_651c_4ccc_89a2_70a03290b1d7.slice/crio-49c3016a0c10868cab86543c95ddff84d91170d856fa62d0a24d02c6bec405ec WatchSource:0}: Error finding container 49c3016a0c10868cab86543c95ddff84d91170d856fa62d0a24d02c6bec405ec: Status 404 returned error can't find the container with id 49c3016a0c10868cab86543c95ddff84d91170d856fa62d0a24d02c6bec405ec
Oct 30 13:09:53.578596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.578571    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"]
Oct 30 13:09:53.669678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.669648    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"]
Oct 30 13:09:53.670271 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:09:53.670251    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaeba70d_8acf_4150_83e5_9f95d6832a1a.slice/crio-fcc1fdd9a23c0aae63b31af9e372bf524e58139d2032929752cc1c2a767282bc WatchSource:0}: Error finding container fcc1fdd9a23c0aae63b31af9e372bf524e58139d2032929752cc1c2a767282bc: Status 404 returned error can't find the container with id fcc1fdd9a23c0aae63b31af9e372bf524e58139d2032929752cc1c2a767282bc
Oct 30 13:09:53.700907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.700852    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8" podUID="753a4ac6-b69b-4330-8b36-a11955e43912" containerName="payload-processing" containerID="cri-o://9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9" gracePeriod=25
Oct 30 13:09:53.734177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:53.734132    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9" podUID="c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0" containerName="payload-pre-processing" containerID="cri-o://cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972" gracePeriod=25
Oct 30 13:09:54.016498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.016469    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:09:54.055845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.055818    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:09:54.124031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.123999    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-plugins-config-volume\") pod \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\" (UID: \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\") "
Oct 30 13:09:54.124216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.124040    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753a4ac6-b69b-4330-8b36-a11955e43912-kube-api-access-4mdps\" (UniqueName: \"kubernetes.io/projected/753a4ac6-b69b-4330-8b36-a11955e43912-kube-api-access-4mdps\") pod \"753a4ac6-b69b-4330-8b36-a11955e43912\" (UID: \"753a4ac6-b69b-4330-8b36-a11955e43912\") "
Oct 30 13:09:54.124320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.124303    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/753a4ac6-b69b-4330-8b36-a11955e43912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/753a4ac6-b69b-4330-8b36-a11955e43912-plugins-config-volume\") pod \"753a4ac6-b69b-4330-8b36-a11955e43912\" (UID: \"753a4ac6-b69b-4330-8b36-a11955e43912\") "
Oct 30 13:09:54.124440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.124398    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-kube-api-access-7zsdd\" (UniqueName: \"kubernetes.io/projected/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-kube-api-access-7zsdd\") pod \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\" (UID: \"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0\") "
Oct 30 13:09:54.124746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.124714    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-plugins-config-volume" pod "c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0" (UID: "c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:54.124808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.124718    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753a4ac6-b69b-4330-8b36-a11955e43912-plugins-config-volume" pod "753a4ac6-b69b-4330-8b36-a11955e43912" (UID: "753a4ac6-b69b-4330-8b36-a11955e43912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:54.125195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.125094    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/753a4ac6-b69b-4330-8b36-a11955e43912-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:54.125195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.125121    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:54.127176 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.127152    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-kube-api-access-7zsdd" pod "c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0" (UID: "c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0"). InnerVolumeSpecName "kube-api-access-7zsdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:54.127383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.127366    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753a4ac6-b69b-4330-8b36-a11955e43912-kube-api-access-4mdps" pod "753a4ac6-b69b-4330-8b36-a11955e43912" (UID: "753a4ac6-b69b-4330-8b36-a11955e43912"). InnerVolumeSpecName "kube-api-access-4mdps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:54.223550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.223506    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq" event={"ID":"aaeba70d-8acf-4150-83e5-9f95d6832a1a","Type":"ContainerStarted","Data":"5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8"}
Oct 30 13:09:54.223747 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.223603    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq" event={"ID":"aaeba70d-8acf-4150-83e5-9f95d6832a1a","Type":"ContainerStarted","Data":"fcc1fdd9a23c0aae63b31af9e372bf524e58139d2032929752cc1c2a767282bc"}
Oct 30 13:09:54.223747 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.223731    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:09:54.225443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.225410    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58" event={"ID":"beaf5b45-651c-4ccc-89a2-70a03290b1d7","Type":"ContainerStarted","Data":"3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700"}
Oct 30 13:09:54.225591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.225464    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58" event={"ID":"beaf5b45-651c-4ccc-89a2-70a03290b1d7","Type":"ContainerStarted","Data":"49c3016a0c10868cab86543c95ddff84d91170d856fa62d0a24d02c6bec405ec"}
Oct 30 13:09:54.225591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.225519    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:09:54.226335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.226276    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mdps\" (UniqueName: \"kubernetes.io/projected/753a4ac6-b69b-4330-8b36-a11955e43912-kube-api-access-4mdps\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:54.226440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.226339    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zsdd\" (UniqueName: \"kubernetes.io/projected/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0-kube-api-access-7zsdd\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:09:54.227161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.227137    4429 generic.go:356] "Generic (PLEG): container finished" podID="753a4ac6-b69b-4330-8b36-a11955e43912" containerID="9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9" exitCode=0
Oct 30 13:09:54.227237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.227189    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"
Oct 30 13:09:54.227237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.227212    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8" event={"ID":"753a4ac6-b69b-4330-8b36-a11955e43912","Type":"ContainerDied","Data":"9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9"}
Oct 30 13:09:54.227369 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.227256    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8" event={"ID":"753a4ac6-b69b-4330-8b36-a11955e43912","Type":"ContainerDied","Data":"eaa14ddaf13d013f3b2bd43572e851c3cd3d804fd404977ade772e4db8649858"}
Oct 30 13:09:54.227369 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.227329    4429 scope.go:122] "RemoveContainer" containerID="9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9"
Oct 30 13:09:54.228859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.228834    4429 generic.go:356] "Generic (PLEG): container finished" podID="c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0" containerID="cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972" exitCode=0
Oct 30 13:09:54.228965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.228875    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9" event={"ID":"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0","Type":"ContainerDied","Data":"cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972"}
Oct 30 13:09:54.228965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.228924    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9" event={"ID":"c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0","Type":"ContainerDied","Data":"a639f1f7fdc73bbf88f05247b15f571d264b23b5c0e6a5a4dd66d9462d3e2311"}
Oct 30 13:09:54.228965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.228931    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"
Oct 30 13:09:54.239258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.239233    4429 scope.go:122] "RemoveContainer" containerID="9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9"
Oct 30 13:09:54.239681 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:09:54.239657    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9\": container with ID starting with 9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9 not found: ID does not exist" containerID="9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9"
Oct 30 13:09:54.239790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.239701    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9"} err="failed to get container status \"9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9\": rpc error: code = NotFound desc = could not find container \"9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9\": container with ID starting with 9c5e5247163cd4703f00f39233bbeabb8e44dbde0124c28f014c084edd6342d9 not found: ID does not exist"
Oct 30 13:09:54.239790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.239727    4429 scope.go:122] "RemoveContainer" containerID="cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972"
Oct 30 13:09:54.250646 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.250623    4429 scope.go:122] "RemoveContainer" containerID="cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972"
Oct 30 13:09:54.251027 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:09:54.251004    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972\": container with ID starting with cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972 not found: ID does not exist" containerID="cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972"
Oct 30 13:09:54.251110 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.251040    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972"} err="failed to get container status \"cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972\": rpc error: code = NotFound desc = could not find container \"cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972\": container with ID starting with cd346b9c78d8cf3aa0ac099d2d3fc384bb1318b6b24ebac0a90cf21f4d497972 not found: ID does not exist"
Oct 30 13:09:54.270146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.270082    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq" podStartSLOduration=2.270064869 podStartE2EDuration="2.270064869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:54.268853674 +0000 UTC m=+2252.324347490" watchObservedRunningTime="2026-10-30 13:09:54.270064869 +0000 UTC m=+2252.325558690"
Oct 30 13:09:54.315885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.315817    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58" podStartSLOduration=2.315802126 podStartE2EDuration="2.315802126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:54.313686955 +0000 UTC m=+2252.369180780" watchObservedRunningTime="2026-10-30 13:09:54.315802126 +0000 UTC m=+2252.371295950"
Oct 30 13:09:54.357381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.357348    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"]
Oct 30 13:09:54.360429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.360393    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f9ea8f-664d6dddc9-8s6s9"]
Oct 30 13:09:54.400202 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.400169    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"]
Oct 30 13:09:54.403365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:54.403338    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f9ea8f-86488b5698-727l8"]
Oct 30 13:09:55.030081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.030045    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"]
Oct 30 13:09:55.030709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.030693    4429 cpu_manager.go:414] "containerMap: removing container" podUID="753a4ac6-b69b-4330-8b36-a11955e43912" containerName="payload-processing"
Oct 30 13:09:55.030756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.030711    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753a4ac6-b69b-4330-8b36-a11955e43912" containerName="payload-processing"
Oct 30 13:09:55.030756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.030729    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0" containerName="payload-pre-processing"
Oct 30 13:09:55.030756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.030735    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0" containerName="payload-pre-processing"
Oct 30 13:09:55.030848 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.030842    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="753a4ac6-b69b-4330-8b36-a11955e43912" containerName="payload-processing"
Oct 30 13:09:55.030881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.030853    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0" containerName="payload-pre-processing"
Oct 30 13:09:55.035952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.035929    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.070205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.070152    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f9c553-openshift-default-dockercfg-5zvwc\""
Oct 30 13:09:55.074075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.074042    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"]
Oct 30 13:09:55.139343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.139297    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istiod-ca-cert\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.139541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.139383    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-data\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.139728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.139692    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-certs\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.139874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.139779    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.139960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.139936    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhw5h\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-kube-api-access-bhw5h\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.140010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.139972    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-envoy\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.140082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.140064    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-token\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.140145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.140130    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-credential-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.140194 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.140155    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-podinfo\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.240918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.240886    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-certs\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.240924    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.240980    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhw5h\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-kube-api-access-bhw5h\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.240998    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-envoy\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241044    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-token\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241099    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-credential-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241125    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-podinfo\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241185    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istiod-ca-cert\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241228    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-data\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241396    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-certs\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241435    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241824    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-credential-socket\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241920    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-data\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.241972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.241962    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istiod-ca-cert\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.243792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.243769    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-envoy\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.243900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.243803    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-podinfo\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.268152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.268122    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-token\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.268328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.268160    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhw5h\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-kube-api-access-bhw5h\") pod \"e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") " pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.352196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.350943    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:55.681831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.681803    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"]
Oct 30 13:09:55.682798 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:09:55.682773    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbd3ed2_7a4b_4191_bc84_67042b85dc8b.slice/crio-f689ebd1ec53cde7d0cdb7c12f23842a65d63ea43e8b74c9a6c948ded8b78e45 WatchSource:0}: Error finding container f689ebd1ec53cde7d0cdb7c12f23842a65d63ea43e8b74c9a6c948ded8b78e45: Status 404 returned error can't find the container with id f689ebd1ec53cde7d0cdb7c12f23842a65d63ea43e8b74c9a6c948ded8b78e45
Oct 30 13:09:55.685751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.685713    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:55.685867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.685811    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:55.685905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:55.685862    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:56.077222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:56.077189    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="753a4ac6-b69b-4330-8b36-a11955e43912" path="/var/lib/kubelet/pods/753a4ac6-b69b-4330-8b36-a11955e43912/volumes"
Oct 30 13:09:56.077689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:56.077675    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0" path="/var/lib/kubelet/pods/c4a70b5d-aac3-4e2d-bf60-1dc40efb31f0/volumes"
Oct 30 13:09:56.246268 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:56.246227    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" event={"ID":"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b","Type":"ContainerStarted","Data":"558233b11abc6c7364546565a536b01e613637152bc450af142a8735bf3e1ec7"}
Oct 30 13:09:56.246467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:56.246329    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" event={"ID":"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b","Type":"ContainerStarted","Data":"f689ebd1ec53cde7d0cdb7c12f23842a65d63ea43e8b74c9a6c948ded8b78e45"}
Oct 30 13:09:56.297140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:56.297076    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podStartSLOduration=1.2970614249999999 podStartE2EDuration="1.297061425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:56.294761756 +0000 UTC m=+2254.350255580" watchObservedRunningTime="2026-10-30 13:09:56.297061425 +0000 UTC m=+2254.352555249"
Oct 30 13:09:56.351896 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:56.351812    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:09:56.354062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:56.354031    4429 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:09:56.354171 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:56.354093    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:09:57.366331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:57.360841    4429 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:09:57.366331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:57.360927    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:09:58.365320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:58.359468    4429 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:09:58.365320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:58.359544    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:09:59.369312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:59.354765    4429 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:09:59.369312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:09:59.354846    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:10:00.365813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:00.365769    4429 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:10:00.366168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:00.366132    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:10:01.372340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:01.370777    4429 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:10:01.372340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:01.370842    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:10:02.352651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:02.352608    4429 patch_prober.go:29] interesting pod/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 13:10:02.352821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:02.352703    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 13:10:03.132953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.132910    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"]
Oct 30 13:10:03.133483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.133200    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq" podUID="bc9601a9-7990-453b-976e-f952b1686868" containerName="maas-api" containerID="cri-o://b09c825b88571019904d34b60b91402932d98928d1d116c4b9274a4eda25f9d2" gracePeriod=30
Oct 30 13:10:03.334953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.334913    4429 generic.go:356] "Generic (PLEG): container finished" podID="bc9601a9-7990-453b-976e-f952b1686868" containerID="b09c825b88571019904d34b60b91402932d98928d1d116c4b9274a4eda25f9d2" exitCode=0
Oct 30 13:10:03.335147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.334993    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq" event={"ID":"bc9601a9-7990-453b-976e-f952b1686868","Type":"ContainerDied","Data":"b09c825b88571019904d34b60b91402932d98928d1d116c4b9274a4eda25f9d2"}
Oct 30 13:10:03.357913 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.357878    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:10:03.358332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.358245    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:10:03.359525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.359503    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:10:03.440248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.440218    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:10:03.472748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.472710    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9601a9-7990-453b-976e-f952b1686868-kube-api-access-mjxvc\" (UniqueName: \"kubernetes.io/projected/bc9601a9-7990-453b-976e-f952b1686868-kube-api-access-mjxvc\") pod \"bc9601a9-7990-453b-976e-f952b1686868\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") "
Oct 30 13:10:03.472946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.472774    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-maas-api-tls\") pod \"bc9601a9-7990-453b-976e-f952b1686868\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") "
Oct 30 13:10:03.472946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.472805    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-metrics-certs\") pod \"bc9601a9-7990-453b-976e-f952b1686868\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") "
Oct 30 13:10:03.473828 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.473156    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9601a9-7990-453b-976e-f952b1686868-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9601a9-7990-453b-976e-f952b1686868-openshift-service-ca\") pod \"bc9601a9-7990-453b-976e-f952b1686868\" (UID: \"bc9601a9-7990-453b-976e-f952b1686868\") "
Oct 30 13:10:03.474654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.474627    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9601a9-7990-453b-976e-f952b1686868-openshift-service-ca" pod "bc9601a9-7990-453b-976e-f952b1686868" (UID: "bc9601a9-7990-453b-976e-f952b1686868"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:03.478886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.476157    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-maas-api-tls" pod "bc9601a9-7990-453b-976e-f952b1686868" (UID: "bc9601a9-7990-453b-976e-f952b1686868"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:03.478886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.476221    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9601a9-7990-453b-976e-f952b1686868-kube-api-access-mjxvc" pod "bc9601a9-7990-453b-976e-f952b1686868" (UID: "bc9601a9-7990-453b-976e-f952b1686868"). InnerVolumeSpecName "kube-api-access-mjxvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:03.479215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.479192    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9601a9-7990-453b-976e-f952b1686868-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:03.480936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.480886    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-metrics-certs" pod "bc9601a9-7990-453b-976e-f952b1686868" (UID: "bc9601a9-7990-453b-976e-f952b1686868"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:03.580922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.580517    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjxvc\" (UniqueName: \"kubernetes.io/projected/bc9601a9-7990-453b-976e-f952b1686868-kube-api-access-mjxvc\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:03.580922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.580550    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:03.580922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:03.580564    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9601a9-7990-453b-976e-f952b1686868-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:04.342844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:04.342756    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"
Oct 30 13:10:04.342844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:04.342796    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq" event={"ID":"bc9601a9-7990-453b-976e-f952b1686868","Type":"ContainerDied","Data":"5d04218a5da13247fa2b6101ae39fb7246189cfb1dfd7faa64c5b52481aa7996"}
Oct 30 13:10:04.343382 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:04.342862    4429 scope.go:122] "RemoveContainer" containerID="b09c825b88571019904d34b60b91402932d98928d1d116c4b9274a4eda25f9d2"
Oct 30 13:10:04.417503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:04.417471    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"]
Oct 30 13:10:04.429549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:04.429518    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f9ea8f-968fc648-9svtq"]
Oct 30 13:10:05.241254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:05.241224    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:10:05.241730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:05.241628    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:10:06.084707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:06.084667    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9601a9-7990-453b-976e-f952b1686868" path="/var/lib/kubelet/pods/bc9601a9-7990-453b-976e-f952b1686868/volumes"
Oct 30 13:10:07.636241 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.636199    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"]
Oct 30 13:10:07.637090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.637068    4429 cpu_manager.go:414] "containerMap: removing container" podUID="bc9601a9-7990-453b-976e-f952b1686868" containerName="maas-api"
Oct 30 13:10:07.637217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.637092    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9601a9-7990-453b-976e-f952b1686868" containerName="maas-api"
Oct 30 13:10:07.637324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.637310    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9601a9-7990-453b-976e-f952b1686868" containerName="maas-api"
Oct 30 13:10:07.642210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.642189    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.665216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.665185    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"]
Oct 30 13:10:07.666069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.666042    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ffbba3-serving-cert\""
Oct 30 13:10:07.741061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.741019    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvk9\" (UniqueName: \"kubernetes.io/projected/d264e49b-9129-42c0-98ee-6bcf79461e8b-kube-api-access-5zvk9\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.741230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.741181    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.741334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.741279    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.741417 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.741405    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d264e49b-9129-42c0-98ee-6bcf79461e8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.843063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.843026    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d264e49b-9129-42c0-98ee-6bcf79461e8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.843248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.843132    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvk9\" (UniqueName: \"kubernetes.io/projected/d264e49b-9129-42c0-98ee-6bcf79461e8b-kube-api-access-5zvk9\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.843248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.843204    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.843365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.843262    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.843492 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:10:07.843472    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-serving-cert: secret "maas-api-e2e-praxis-b-ffbba3-serving-cert" not found
Oct 30 13:10:07.843580 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:10:07.843568    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls podName:d264e49b-9129-42c0-98ee-6bcf79461e8b nodeName:}" failed. No retries permitted until 2026-10-30 13:10:08.34354445 +0000 UTC m=+2266.399038273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls") pod "maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f" (UID: "d264e49b-9129-42c0-98ee-6bcf79461e8b") : secret "maas-api-e2e-praxis-b-ffbba3-serving-cert" not found
Oct 30 13:10:07.843882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.843863    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d264e49b-9129-42c0-98ee-6bcf79461e8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.845907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.845874    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:07.871532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:07.871500    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvk9\" (UniqueName: \"kubernetes.io/projected/d264e49b-9129-42c0-98ee-6bcf79461e8b-kube-api-access-5zvk9\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:08.350495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.350451    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:08.353113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.353083    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:08.572971 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.572933    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:08.572971 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.572948    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"]
Oct 30 13:10:08.577365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.577332    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.606829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.606743    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"]
Oct 30 13:10:08.656697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.656643    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.657209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.656717    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.657209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.656794    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.657209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.656818    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.657209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.656863    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.657209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.656912    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.657209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.656961    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.657732 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.657247    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.657732 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.657358    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxbv\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-kube-api-access-ptxbv\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.768877    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.768936    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxbv\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-kube-api-access-ptxbv\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.768980    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.768997    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.769036    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.769051    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.769080    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.769112    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.769141    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.769624    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.770321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.769636    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.777577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.770411    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.777577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.771051    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.777577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.771408    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.778713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.778603    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.784183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.784073    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.797158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.797120    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.799246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.799211    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxbv\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-kube-api-access-ptxbv\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:08.914384 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:10:08.914338    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd264e49b_9129_42c0_98ee_6bcf79461e8b.slice/crio-92a54d9745bad376fc58c8dedbe2d0c1526a9332c7d416eba6bdbf0f23da72ac WatchSource:0}: Error finding container 92a54d9745bad376fc58c8dedbe2d0c1526a9332c7d416eba6bdbf0f23da72ac: Status 404 returned error can't find the container with id 92a54d9745bad376fc58c8dedbe2d0c1526a9332c7d416eba6bdbf0f23da72ac
Oct 30 13:10:08.915006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.914982    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"]
Oct 30 13:10:08.931015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:08.930985    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:09.260179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.260158    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"]
Oct 30 13:10:09.260486 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:10:09.260456    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5e28f6_11d7_496c_acee_cc70e26a9be7.slice/crio-70ced8f2c2547c7a89f6a7914ce0fa1647de76c5a64e539255e7d596c6a1f33a WatchSource:0}: Error finding container 70ced8f2c2547c7a89f6a7914ce0fa1647de76c5a64e539255e7d596c6a1f33a: Status 404 returned error can't find the container with id 70ced8f2c2547c7a89f6a7914ce0fa1647de76c5a64e539255e7d596c6a1f33a
Oct 30 13:10:09.263512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.263481    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:09.263601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.263556    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:09.263601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.263588    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:09.386685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.386645    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f" event={"ID":"d264e49b-9129-42c0-98ee-6bcf79461e8b","Type":"ContainerStarted","Data":"04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b"}
Oct 30 13:10:09.386864 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.386706    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f" event={"ID":"d264e49b-9129-42c0-98ee-6bcf79461e8b","Type":"ContainerStarted","Data":"92a54d9745bad376fc58c8dedbe2d0c1526a9332c7d416eba6bdbf0f23da72ac"}
Oct 30 13:10:09.386864 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.386755    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:09.391395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.391343    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" event={"ID":"3a5e28f6-11d7-496c-acee-cc70e26a9be7","Type":"ContainerStarted","Data":"adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761"}
Oct 30 13:10:09.391532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.391414    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" event={"ID":"3a5e28f6-11d7-496c-acee-cc70e26a9be7","Type":"ContainerStarted","Data":"70ced8f2c2547c7a89f6a7914ce0fa1647de76c5a64e539255e7d596c6a1f33a"}
Oct 30 13:10:09.436791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.436741    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f" podStartSLOduration=2.14705059 podStartE2EDuration="2.436726076s" totalImagesPullingTime="289.675486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:09.432752631 +0000 UTC m=+2267.488246465" watchObservedRunningTime="2026-10-30 13:10:09.436726076 +0000 UTC m=+2267.492219900"
Oct 30 13:10:09.483490 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.483374    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podStartSLOduration=1.483356558 podStartE2EDuration="1.483356558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:09.482750796 +0000 UTC m=+2267.538244641" watchObservedRunningTime="2026-10-30 13:10:09.483356558 +0000 UTC m=+2267.538850413"
Oct 30 13:10:09.931538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:09.931471    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:10.876873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:10.876839    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"]
Oct 30 13:10:10.877138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:10.877100    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2" podUID="2a1b7e09-67f1-4df1-b385-ae9ae8370634" containerName="payload-processing" containerID="cri-o://9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4" gracePeriod=30
Oct 30 13:10:10.933191 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:10.932944    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:10:10.933191 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:10.933006    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:10:11.239528 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.239498    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:10:11.323309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.322822    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1b7e09-67f1-4df1-b385-ae9ae8370634-kube-api-access-bnll5\" (UniqueName: \"kubernetes.io/projected/2a1b7e09-67f1-4df1-b385-ae9ae8370634-kube-api-access-bnll5\") pod \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\" (UID: \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\") "
Oct 30 13:10:11.323309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.322908    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1b7e09-67f1-4df1-b385-ae9ae8370634-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b7e09-67f1-4df1-b385-ae9ae8370634-plugins-config-volume\") pod \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\" (UID: \"2a1b7e09-67f1-4df1-b385-ae9ae8370634\") "
Oct 30 13:10:11.324123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.324090    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1b7e09-67f1-4df1-b385-ae9ae8370634-plugins-config-volume" pod "2a1b7e09-67f1-4df1-b385-ae9ae8370634" (UID: "2a1b7e09-67f1-4df1-b385-ae9ae8370634"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:11.341521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.341483    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1b7e09-67f1-4df1-b385-ae9ae8370634-kube-api-access-bnll5" pod "2a1b7e09-67f1-4df1-b385-ae9ae8370634" (UID: "2a1b7e09-67f1-4df1-b385-ae9ae8370634"). InnerVolumeSpecName "kube-api-access-bnll5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:11.415238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.415193    4429 generic.go:356] "Generic (PLEG): container finished" podID="2a1b7e09-67f1-4df1-b385-ae9ae8370634" containerID="9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4" exitCode=0
Oct 30 13:10:11.416000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.415973    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"
Oct 30 13:10:11.416421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.416364    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2" event={"ID":"2a1b7e09-67f1-4df1-b385-ae9ae8370634","Type":"ContainerDied","Data":"9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4"}
Oct 30 13:10:11.416421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.416413    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2" event={"ID":"2a1b7e09-67f1-4df1-b385-ae9ae8370634","Type":"ContainerDied","Data":"ebce32220279c30fb9a69763bb6971a6bae5cff8d71d33167f07e539deee91eb"}
Oct 30 13:10:11.416602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.416444    4429 scope.go:122] "RemoveContainer" containerID="9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4"
Oct 30 13:10:11.424695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.424672    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnll5\" (UniqueName: \"kubernetes.io/projected/2a1b7e09-67f1-4df1-b385-ae9ae8370634-kube-api-access-bnll5\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:11.426252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.426235    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b7e09-67f1-4df1-b385-ae9ae8370634-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:11.435535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.435515    4429 scope.go:122] "RemoveContainer" containerID="9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4"
Oct 30 13:10:11.435802 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:10:11.435782    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4\": container with ID starting with 9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4 not found: ID does not exist" containerID="9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4"
Oct 30 13:10:11.435852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.435812    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4"} err="failed to get container status \"9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4\": rpc error: code = NotFound desc = could not find container \"9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4\": container with ID starting with 9d8bb6ab33cb17f6f96bc2bdae1ee95357f76d54d0bd3af7eee01e5ff56ebfa4 not found: ID does not exist"
Oct 30 13:10:11.499670 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.499584    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"]
Oct 30 13:10:11.526906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.526875    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f9c553-7b94f54fb-fvwz2"]
Oct 30 13:10:11.931766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.931688    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:10:11.931766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:11.931741    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:10:12.084832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:12.084796    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1b7e09-67f1-4df1-b385-ae9ae8370634" path="/var/lib/kubelet/pods/2a1b7e09-67f1-4df1-b385-ae9ae8370634/volumes"
Oct 30 13:10:12.434537 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:10:12.434505    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:10:12.931909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:12.931826    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:10:12.931909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:12.931886    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:10:13.088879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.088834    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"]
Oct 30 13:10:13.089822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.089799    4429 cpu_manager.go:414] "containerMap: removing container" podUID="2a1b7e09-67f1-4df1-b385-ae9ae8370634" containerName="payload-processing"
Oct 30 13:10:13.089970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.089824    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1b7e09-67f1-4df1-b385-ae9ae8370634" containerName="payload-processing"
Oct 30 13:10:13.090052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.089986    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1b7e09-67f1-4df1-b385-ae9ae8370634" containerName="payload-processing"
Oct 30 13:10:13.096780 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.096756    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.115014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.114988    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"]
Oct 30 13:10:13.175222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175188    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.175414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175274    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d631b85a-dadd-4bfa-8460-7955363b5824-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.175414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175347    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.175414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175364    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmf6s\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-kube-api-access-rmf6s\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.175414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175409    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d631b85a-dadd-4bfa-8460-7955363b5824-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.175678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175500    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.175678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175526    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.175678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175550    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.175678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.175583    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276446    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276550    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d631b85a-dadd-4bfa-8460-7955363b5824-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276607    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276633    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmf6s\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-kube-api-access-rmf6s\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276675    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d631b85a-dadd-4bfa-8460-7955363b5824-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276791    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276816    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276844    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.276949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.276887    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.277180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.277115    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-credential-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.277346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.277322    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-socket\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.277935 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.277906    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-certs\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.278058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.278007    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d631b85a-dadd-4bfa-8460-7955363b5824-istiod-ca-cert\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.278139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.278123    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-data\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.280151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.280125    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-envoy\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.280354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.280333    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d631b85a-dadd-4bfa-8460-7955363b5824-istio-podinfo\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.313811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.313786    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-istio-token\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.313811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.313899    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmf6s\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-kube-api-access-rmf6s\") pod \"e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") " pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.416696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.416671    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:13.776198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.776121    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"]
Oct 30 13:10:13.779026 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:10:13.778984    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd631b85a_dadd_4bfa_8460_7955363b5824.slice/crio-4c39cff382b295e97ec4a96898a62a65eb4e804c903ae7aa1eb515eef0d21e2f WatchSource:0}: Error finding container 4c39cff382b295e97ec4a96898a62a65eb4e804c903ae7aa1eb515eef0d21e2f: Status 404 returned error can't find the container with id 4c39cff382b295e97ec4a96898a62a65eb4e804c903ae7aa1eb515eef0d21e2f
Oct 30 13:10:13.781703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.781664    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:13.781787 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.781736    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:13.781787 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.781769    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:13.931726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.931697    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:10:13.931894 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:13.931747    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:10:14.480142 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:14.480107    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" event={"ID":"d631b85a-dadd-4bfa-8460-7955363b5824","Type":"ContainerStarted","Data":"fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae"}
Oct 30 13:10:14.480603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:14.480167    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" event={"ID":"d631b85a-dadd-4bfa-8460-7955363b5824","Type":"ContainerStarted","Data":"4c39cff382b295e97ec4a96898a62a65eb4e804c903ae7aa1eb515eef0d21e2f"}
Oct 30 13:10:14.531174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:14.531126    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podStartSLOduration=1.531112652 podStartE2EDuration="1.531112652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:14.528236693 +0000 UTC m=+2272.583730520" watchObservedRunningTime="2026-10-30 13:10:14.531112652 +0000 UTC m=+2272.586606476"
Oct 30 13:10:14.931749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:14.931663    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:10:14.931749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:14.931717    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:10:15.405987 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:15.405958    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:10:15.417077 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:15.417057    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:15.418378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:15.418339    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:10:15.418476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:15.418421    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:10:15.931820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:15.931779    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:10:15.932210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:15.931839    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 13:10:16.417885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:16.417851    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:10:16.418099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:16.417915    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:10:17.225070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:17.225035    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"]
Oct 30 13:10:17.419343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:17.418645    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:10:17.419343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:17.418707    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:10:17.947465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:17.947432    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:10:17.947768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:17.947735    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:18.421375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:18.420623    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:10:18.421375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:18.420690    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:10:18.485306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:18.484372    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 13:10:18.496311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:18.495229    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:18.648422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:18.648365    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:10:18.940322 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:18.916533    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:18.954426 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:18.951583    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:10:18.954426 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:18.951640    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:10:19.448358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.440939    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:19.448358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.443787    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:10:19.448358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.443840    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:10:19.459567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.459522    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 13:10:19.561526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.561482    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:19.732367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.731209    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:19.768311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.755415    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:19.768311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.756153    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:10:19.778312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.774457    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:19.778312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.775010    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:10:19.823312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.822546    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:10:19.946248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:19.946134    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:10:20.010755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.010653    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:10:20.011034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.010989    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:20.218256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.218221    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:10:20.222428 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.222397    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:10:20.418030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.417946    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:10:20.418336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.418301    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:10:20.454313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.451645    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:20.454313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.453175    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:20.482320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:20.466854    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:10:21.417355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:21.417322    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:10:21.417548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:21.417391    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 13:10:23.314920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:23.314887    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"]
Oct 30 13:10:23.461499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:23.461463    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:10:23.461761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:23.461545    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:24.081114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:24.081077    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:24.081528 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:24.081500    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:24.082777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:24.082754    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:10:28.315662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.315614    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj" podUID="1a66ac8b-b9bd-4335-bb8e-91d102a88280" containerName="payload-pre-processing" containerID="cri-o://150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3" gracePeriod=25
Oct 30 13:10:28.594845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.594818    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:10:28.645373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.645334    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a66ac8b-b9bd-4335-bb8e-91d102a88280-kube-api-access-mcmsg\" (UniqueName: \"kubernetes.io/projected/1a66ac8b-b9bd-4335-bb8e-91d102a88280-kube-api-access-mcmsg\") pod \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\" (UID: \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\") "
Oct 30 13:10:28.645577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.645520    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a66ac8b-b9bd-4335-bb8e-91d102a88280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a66ac8b-b9bd-4335-bb8e-91d102a88280-plugins-config-volume\") pod \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\" (UID: \"1a66ac8b-b9bd-4335-bb8e-91d102a88280\") "
Oct 30 13:10:28.646151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.646110    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a66ac8b-b9bd-4335-bb8e-91d102a88280-plugins-config-volume" pod "1a66ac8b-b9bd-4335-bb8e-91d102a88280" (UID: "1a66ac8b-b9bd-4335-bb8e-91d102a88280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:28.646409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.646381    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a66ac8b-b9bd-4335-bb8e-91d102a88280-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:28.656538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.656495    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a66ac8b-b9bd-4335-bb8e-91d102a88280-kube-api-access-mcmsg" pod "1a66ac8b-b9bd-4335-bb8e-91d102a88280" (UID: "1a66ac8b-b9bd-4335-bb8e-91d102a88280"). InnerVolumeSpecName "kube-api-access-mcmsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:28.747821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.747779    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcmsg\" (UniqueName: \"kubernetes.io/projected/1a66ac8b-b9bd-4335-bb8e-91d102a88280-kube-api-access-mcmsg\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:28.989235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.989188    4429 generic.go:356] "Generic (PLEG): container finished" podID="1a66ac8b-b9bd-4335-bb8e-91d102a88280" containerID="150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3" exitCode=0
Oct 30 13:10:28.989461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.989250    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj" event={"ID":"1a66ac8b-b9bd-4335-bb8e-91d102a88280","Type":"ContainerDied","Data":"150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3"}
Oct 30 13:10:28.989461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.989258    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"
Oct 30 13:10:28.989461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.989357    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj" event={"ID":"1a66ac8b-b9bd-4335-bb8e-91d102a88280","Type":"ContainerDied","Data":"40d01b47b84e219c0028b4af23392391b5a7da1ed577a30b84119eebce92e9ed"}
Oct 30 13:10:28.989461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:28.989384    4429 scope.go:122] "RemoveContainer" containerID="150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3"
Oct 30 13:10:29.020216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:29.020190    4429 scope.go:122] "RemoveContainer" containerID="150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3"
Oct 30 13:10:29.020710 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:10:29.020686    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3\": container with ID starting with 150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3 not found: ID does not exist" containerID="150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3"
Oct 30 13:10:29.020813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:29.020723    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3"} err="failed to get container status \"150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3\": rpc error: code = NotFound desc = could not find container \"150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3\": container with ID starting with 150a0d3a308f18ff21848a7ee7b5c920cb19942434c0de88ac922da06393d3e3 not found: ID does not exist"
Oct 30 13:10:29.060738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:29.060705    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"]
Oct 30 13:10:29.063956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:29.063925    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d84f47-7d4688cc47-8wdvj"]
Oct 30 13:10:30.076230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:30.076182    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a66ac8b-b9bd-4335-bb8e-91d102a88280" path="/var/lib/kubelet/pods/1a66ac8b-b9bd-4335-bb8e-91d102a88280/volumes"
Oct 30 13:10:32.320405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.320363    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf" podUID="95ca8a62-2b26-46d5-8d14-94aadc6ef68b" containerName="main" containerID="cri-o://a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6" gracePeriod=45
Oct 30 13:10:32.609549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.609520    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:10:32.713068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.713015    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tmp-dir\") pod \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") "
Oct 30 13:10:32.713319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.713195    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-dshm\") pod \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") "
Oct 30 13:10:32.713398 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.713333    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-kube-api-access-wfnqx\" (UniqueName: \"kubernetes.io/projected/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-kube-api-access-wfnqx\") pod \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") "
Oct 30 13:10:32.713449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.713380    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tmp-dir" pod "95ca8a62-2b26-46d5-8d14-94aadc6ef68b" (UID: "95ca8a62-2b26-46d5-8d14-94aadc6ef68b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.713850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.713822    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-home\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-home\") pod \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") "
Oct 30 13:10:32.714147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.714095    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-home" pod "95ca8a62-2b26-46d5-8d14-94aadc6ef68b" (UID: "95ca8a62-2b26-46d5-8d14-94aadc6ef68b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.714271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.714252    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tls-certs\" (UniqueName: \"kubernetes.io/secret/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tls-certs\") pod \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") "
Oct 30 13:10:32.714704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.714663    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-model-cache\") pod \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\" (UID: \"95ca8a62-2b26-46d5-8d14-94aadc6ef68b\") "
Oct 30 13:10:32.714957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.714932    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-model-cache" pod "95ca8a62-2b26-46d5-8d14-94aadc6ef68b" (UID: "95ca8a62-2b26-46d5-8d14-94aadc6ef68b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.715663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.715638    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.715792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.715668    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.715792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.715680    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.717062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.716124    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-dshm" pod "95ca8a62-2b26-46d5-8d14-94aadc6ef68b" (UID: "95ca8a62-2b26-46d5-8d14-94aadc6ef68b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:32.717062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.716600    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-kube-api-access-wfnqx" pod "95ca8a62-2b26-46d5-8d14-94aadc6ef68b" (UID: "95ca8a62-2b26-46d5-8d14-94aadc6ef68b"). InnerVolumeSpecName "kube-api-access-wfnqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:32.718614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.718583    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tls-certs" pod "95ca8a62-2b26-46d5-8d14-94aadc6ef68b" (UID: "95ca8a62-2b26-46d5-8d14-94aadc6ef68b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:32.816725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.816692    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.816725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.816721    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnqx\" (UniqueName: \"kubernetes.io/projected/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-kube-api-access-wfnqx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:32.816725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:32.816732    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ca8a62-2b26-46d5-8d14-94aadc6ef68b-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:10:33.020629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.020595    4429 generic.go:356] "Generic (PLEG): container finished" podID="95ca8a62-2b26-46d5-8d14-94aadc6ef68b" containerID="a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6" exitCode=0
Oct 30 13:10:33.020835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.020649    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"
Oct 30 13:10:33.020835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.020688    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf" event={"ID":"95ca8a62-2b26-46d5-8d14-94aadc6ef68b","Type":"ContainerDied","Data":"a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6"}
Oct 30 13:10:33.020835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.020739    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf" event={"ID":"95ca8a62-2b26-46d5-8d14-94aadc6ef68b","Type":"ContainerDied","Data":"160733c493b57f9a2cfdab47f3056dbf0b64554e9252895327d03eb3fbbb160a"}
Oct 30 13:10:33.020835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.020761    4429 scope.go:122] "RemoveContainer" containerID="a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6"
Oct 30 13:10:33.030886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.030863    4429 scope.go:122] "RemoveContainer" containerID="a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6"
Oct 30 13:10:33.031150 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:10:33.031132    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6\": container with ID starting with a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6 not found: ID does not exist" containerID="a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6"
Oct 30 13:10:33.031216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.031162    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6"} err="failed to get container status \"a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6\": rpc error: code = NotFound desc = could not find container \"a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6\": container with ID starting with a617277a5db1f4de4cf20dae5368bf37e947c1b3af46e51a3c84cea73d5037f6 not found: ID does not exist"
Oct 30 13:10:33.083114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.083080    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"]
Oct 30 13:10:33.097834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:33.097801    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f9c553/ipp-route-f9c553-kserve-574485b6bf-rg5tf"]
Oct 30 13:10:34.076131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:34.076101    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ca8a62-2b26-46d5-8d14-94aadc6ef68b" path="/var/lib/kubelet/pods/95ca8a62-2b26-46d5-8d14-94aadc6ef68b/volumes"
Oct 30 13:10:38.786822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.786791    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"]
Oct 30 13:10:38.787439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.787423    4429 cpu_manager.go:414] "containerMap: removing container" podUID="95ca8a62-2b26-46d5-8d14-94aadc6ef68b" containerName="main"
Oct 30 13:10:38.787512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.787439    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ca8a62-2b26-46d5-8d14-94aadc6ef68b" containerName="main"
Oct 30 13:10:38.787512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.787462    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1a66ac8b-b9bd-4335-bb8e-91d102a88280" containerName="payload-pre-processing"
Oct 30 13:10:38.787512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.787468    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a66ac8b-b9bd-4335-bb8e-91d102a88280" containerName="payload-pre-processing"
Oct 30 13:10:38.787616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.787546    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ca8a62-2b26-46d5-8d14-94aadc6ef68b" containerName="main"
Oct 30 13:10:38.787616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.787558    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a66ac8b-b9bd-4335-bb8e-91d102a88280" containerName="payload-pre-processing"
Oct 30 13:10:38.792151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.792135    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805320    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805426    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3069c919-876a-4e81-89c2-05a39f1b4f51-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805484    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805584 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805570    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805646 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805635    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5llq\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-kube-api-access-w5llq\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805661    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805789 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805769    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805805    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.805872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.805861    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.843963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.843910    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"]
Oct 30 13:10:38.906732 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.906701    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.906868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.906793    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5llq\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-kube-api-access-w5llq\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.906868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.906833    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.906948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.906891    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.906948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.906927    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.906975    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.907004    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.907070    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3069c919-876a-4e81-89c2-05a39f1b4f51-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.907132    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.907155    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-certs\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.907307    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.907373    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-data\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.907549    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-credential-socket\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.907840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.907819    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3069c919-876a-4e81-89c2-05a39f1b4f51-istiod-ca-cert\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.909353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.909330    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-envoy\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.909622 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.909603    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-podinfo\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.934179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.934113    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5llq\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-kube-api-access-w5llq\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:38.941210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:38.941180    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-token\") pod \"e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") " pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:39.105602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:39.105508    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:39.428972 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:10:39.428942    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3069c919_876a_4e81_89c2_05a39f1b4f51.slice/crio-7b9c61a9c76465175e4e04a5df8fb5b933cb357935620000850c60b90fb53320 WatchSource:0}: Error finding container 7b9c61a9c76465175e4e04a5df8fb5b933cb357935620000850c60b90fb53320: Status 404 returned error can't find the container with id 7b9c61a9c76465175e4e04a5df8fb5b933cb357935620000850c60b90fb53320
Oct 30 13:10:39.429859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:39.429840    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"]
Oct 30 13:10:39.431733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:39.431683    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:39.431833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:39.431765    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:39.431833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:39.431793    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:40.081409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:40.081377    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" event={"ID":"3069c919-876a-4e81-89c2-05a39f1b4f51","Type":"ContainerStarted","Data":"f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2"}
Oct 30 13:10:40.081790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:40.081433    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" event={"ID":"3069c919-876a-4e81-89c2-05a39f1b4f51","Type":"ContainerStarted","Data":"7b9c61a9c76465175e4e04a5df8fb5b933cb357935620000850c60b90fb53320"}
Oct 30 13:10:40.106413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:40.106389    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:40.108136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:40.108114    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:10:40.108245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:40.108154    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:10:40.177933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:40.176801    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podStartSLOduration=2.176777397 podStartE2EDuration="2.176777397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:40.17366259 +0000 UTC m=+2298.229156417" watchObservedRunningTime="2026-10-30 13:10:40.176777397 +0000 UTC m=+2298.232271222"
Oct 30 13:10:41.106522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:41.106488    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:10:41.106897 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:41.106569    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:10:42.106732 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:42.106699    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:10:42.107130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:42.106743    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:10:42.398855 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:10:42.398770    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:10:43.105965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:43.105931    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:10:43.106139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:43.105986    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:10:44.177306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:44.174624    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:10:44.177306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:44.174688    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:10:45.107254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:45.107210    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:10:45.107548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:45.107506    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:10:45.141385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:45.141341    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:45.162361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:45.162323    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:45.415491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:45.412394    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 13:10:45.415491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:45.414994    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 13:10:46.117315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:46.115343    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:10:46.117315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:46.115411    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:10:46.161412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:46.161362    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 13:10:46.178315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:46.163103    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:46.264311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:46.234011    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:10:46.423367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:46.422390    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:10:46.466874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:46.424050    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:10:47.039316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:47.037915    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:10:47.148464 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:47.145911    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:10:47.149522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:47.149493    4429 patch_prober.go:29] interesting pod/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 13:10:47.196527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:47.196486    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 13:10:48.695886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:48.695853    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:48.696480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:48.696456    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:48.697332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:48.697313    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:10:55.870374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:55.870340    4429 generic.go:356] "Generic (PLEG): container finished" podID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerID="c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794" exitCode=0
Oct 30 13:10:55.870782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:55.870382    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" event={"ID":"1a805ee4-c7f9-4e63-a427-ec946740e9bf","Type":"ContainerDied","Data":"c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794"}
Oct 30 13:10:55.870831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:55.870813    4429 scope.go:122] "RemoveContainer" containerID="c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794"
Oct 30 13:10:55.872439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:55.872411    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:55.872514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:55.872484    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:55.872550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:55.872517    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:56.882757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:56.882340    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" event={"ID":"1a805ee4-c7f9-4e63-a427-ec946740e9bf","Type":"ContainerStarted","Data":"ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef"}
Oct 30 13:10:57.840430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:57.840395    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:10:57.840603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:57.840554    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:10:57.840648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:57.840608    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:10:58.840902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:58.840869    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:10:58.841323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:58.840951    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:10:59.840946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:59.840912    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:10:59.840946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:10:59.840979    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:11:00.840461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:00.840426    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:11:00.840637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:00.840483    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:11:01.840829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:01.840791    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 13:11:01.841245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:01.840870    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 13:11:02.844621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:02.844593    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:11:02.845027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:02.844896    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:11:02.845656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:02.845636    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:11:03.628354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:03.628320    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"]
Oct 30 13:11:03.628666 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:03.628613    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy" containerID="cri-o://558233b11abc6c7364546565a536b01e613637152bc450af142a8735bf3e1ec7" gracePeriod=30
Oct 30 13:11:08.103895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:08.103827    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.129011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:08.128962    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.157361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:08.148246    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.212440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:08.212384    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.399488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:08.398388    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:08.888375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:08.888062    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.147343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.118639    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.147343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.126605    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:11:09.297112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.148962    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.297112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.175081    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:11:09.297112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.175552    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.297112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.175962    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:11:09.297112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.248379    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.297112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.248759    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:11:09.365936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.365445    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 13:11:09.427628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.425029    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:09.427628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.425843    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:11:09.656333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.612863    4429 generic.go:356] "Generic (PLEG): container finished" podID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerID="558233b11abc6c7364546565a536b01e613637152bc450af142a8735bf3e1ec7" exitCode=0
Oct 30 13:11:09.656333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:09.613027    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" event={"ID":"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b","Type":"ContainerDied","Data":"558233b11abc6c7364546565a536b01e613637152bc450af142a8735bf3e1ec7"}
Oct 30 13:11:10.140517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.140471    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 13:11:10.157556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.157516    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 13:11:10.211437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.211371    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:10.217174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.217145    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:11:10.218551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.218534    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:11:10.248613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.248568    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:10.266957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.266934    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:11:10.333341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.332363    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:11:10.364780 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.364750    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:11:10.366386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.366367    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:11:10.430809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.430724    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istiod-ca-cert\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.430989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.430859    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-data\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.430989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.430950    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-token\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.431119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431024    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-certs\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.431119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431076    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-podinfo\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.431231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431110    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istiod-ca-cert" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:10.431231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431139    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-socket\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.431231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431147    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-data" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.431231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431221    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-credential-socket\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.431504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431272    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-kube-api-access-bhw5h\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-kube-api-access-bhw5h\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.431504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431404    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-envoy\") pod \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\" (UID: \"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b\") "
Oct 30 13:11:10.431504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431393    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-certs" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.431504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431470    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-socket" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.431943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431919    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-credential-socket" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.432271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.431960    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.432431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.432333    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.432431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.432351    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.432431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.432364    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.433891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.433864    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-token" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:10.434130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.434110    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-kube-api-access-bhw5h" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "kube-api-access-bhw5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:10.434510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.434489    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-podinfo" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:10.434590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.434553    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-envoy" pod "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" (UID: "6bbd3ed2-7a4b-4191-bc84-67042b85dc8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:10.534207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.534175    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.534207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.534202    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.534511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.534214    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.534511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.534229    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhw5h\" (UniqueName: \"kubernetes.io/projected/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-kube-api-access-bhw5h\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.534511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.534243    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:10.627178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.627146    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"
Oct 30 13:11:10.627178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.627176    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf" event={"ID":"6bbd3ed2-7a4b-4191-bc84-67042b85dc8b","Type":"ContainerDied","Data":"f689ebd1ec53cde7d0cdb7c12f23842a65d63ea43e8b74c9a6c948ded8b78e45"}
Oct 30 13:11:10.627453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.627246    4429 scope.go:122] "RemoveContainer" containerID="558233b11abc6c7364546565a536b01e613637152bc450af142a8735bf3e1ec7"
Oct 30 13:11:10.711074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.711044    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"]
Oct 30 13:11:10.736954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:10.736928    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9c553-openshift-default-77f8888997-fjpwf"]
Oct 30 13:11:12.076372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:12.076340    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" path="/var/lib/kubelet/pods/6bbd3ed2-7a4b-4191-bc84-67042b85dc8b/volumes"
Oct 30 13:11:12.372880 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:11:12.372789    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:11:14.930236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:14.930187    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"]
Oct 30 13:11:14.980193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:14.980160    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"]
Oct 30 13:11:18.061429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.061391    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"]
Oct 30 13:11:18.062044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.061702    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f" podUID="d264e49b-9129-42c0-98ee-6bcf79461e8b" containerName="maas-api" containerID="cri-o://04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b" gracePeriod=30
Oct 30 13:11:18.355463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.355439    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:11:18.468540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.468499    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-metrics-certs\") pod \"d264e49b-9129-42c0-98ee-6bcf79461e8b\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") "
Oct 30 13:11:18.469391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.469364    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls\") pod \"d264e49b-9129-42c0-98ee-6bcf79461e8b\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") "
Oct 30 13:11:18.470469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.470340    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d264e49b-9129-42c0-98ee-6bcf79461e8b-openshift-service-ca" pod "d264e49b-9129-42c0-98ee-6bcf79461e8b" (UID: "d264e49b-9129-42c0-98ee-6bcf79461e8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:18.470627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.470498    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d264e49b-9129-42c0-98ee-6bcf79461e8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d264e49b-9129-42c0-98ee-6bcf79461e8b-openshift-service-ca\") pod \"d264e49b-9129-42c0-98ee-6bcf79461e8b\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") "
Oct 30 13:11:18.470950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.470930    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d264e49b-9129-42c0-98ee-6bcf79461e8b-kube-api-access-5zvk9\" (UniqueName: \"kubernetes.io/projected/d264e49b-9129-42c0-98ee-6bcf79461e8b-kube-api-access-5zvk9\") pod \"d264e49b-9129-42c0-98ee-6bcf79461e8b\" (UID: \"d264e49b-9129-42c0-98ee-6bcf79461e8b\") "
Oct 30 13:11:18.471451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.471425    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-metrics-certs" pod "d264e49b-9129-42c0-98ee-6bcf79461e8b" (UID: "d264e49b-9129-42c0-98ee-6bcf79461e8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:18.472574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.472405    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:18.472574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.472434    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d264e49b-9129-42c0-98ee-6bcf79461e8b-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:18.474657 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.474620    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d264e49b-9129-42c0-98ee-6bcf79461e8b-kube-api-access-5zvk9" pod "d264e49b-9129-42c0-98ee-6bcf79461e8b" (UID: "d264e49b-9129-42c0-98ee-6bcf79461e8b"). InnerVolumeSpecName "kube-api-access-5zvk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:18.474890 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.474861    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls" pod "d264e49b-9129-42c0-98ee-6bcf79461e8b" (UID: "d264e49b-9129-42c0-98ee-6bcf79461e8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:18.521381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.521342    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"]
Oct 30 13:11:18.521626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.521594    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8" podUID="97f8256f-4dee-4d1a-91e6-8cfd197c1362" containerName="maas-api" containerID="cri-o://eda4f335c56e792023d4a62203c60c83ad12881acbfab330924cd40ccdbcee57" gracePeriod=30
Oct 30 13:11:18.573818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.573793    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zvk9\" (UniqueName: \"kubernetes.io/projected/d264e49b-9129-42c0-98ee-6bcf79461e8b-kube-api-access-5zvk9\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:18.573818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.573819    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d264e49b-9129-42c0-98ee-6bcf79461e8b-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:18.684450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.684419    4429 generic.go:356] "Generic (PLEG): container finished" podID="97f8256f-4dee-4d1a-91e6-8cfd197c1362" containerID="eda4f335c56e792023d4a62203c60c83ad12881acbfab330924cd40ccdbcee57" exitCode=0
Oct 30 13:11:18.684640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.684507    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8" event={"ID":"97f8256f-4dee-4d1a-91e6-8cfd197c1362","Type":"ContainerDied","Data":"eda4f335c56e792023d4a62203c60c83ad12881acbfab330924cd40ccdbcee57"}
Oct 30 13:11:18.685911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.685891    4429 generic.go:356] "Generic (PLEG): container finished" podID="d264e49b-9129-42c0-98ee-6bcf79461e8b" containerID="04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b" exitCode=0
Oct 30 13:11:18.686016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.685957    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f" event={"ID":"d264e49b-9129-42c0-98ee-6bcf79461e8b","Type":"ContainerDied","Data":"04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b"}
Oct 30 13:11:18.686016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.685986    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f" event={"ID":"d264e49b-9129-42c0-98ee-6bcf79461e8b","Type":"ContainerDied","Data":"92a54d9745bad376fc58c8dedbe2d0c1526a9332c7d416eba6bdbf0f23da72ac"}
Oct 30 13:11:18.686016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.686007    4429 scope.go:122] "RemoveContainer" containerID="04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b"
Oct 30 13:11:18.686181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.686158    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"
Oct 30 13:11:18.698194 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.698171    4429 scope.go:122] "RemoveContainer" containerID="04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b"
Oct 30 13:11:18.698544 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:11:18.698519    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b\": container with ID starting with 04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b not found: ID does not exist" containerID="04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b"
Oct 30 13:11:18.698636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.698549    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b"} err="failed to get container status \"04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b\": rpc error: code = NotFound desc = could not find container \"04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b\": container with ID starting with 04c3fd6ca5d1f2a18e3ce43b28d122fa14e2a4495e5a7770b176174b3ade8b2b not found: ID does not exist"
Oct 30 13:11:18.719206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.719180    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"]
Oct 30 13:11:18.719476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.719434    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2" podUID="188768bd-7283-4c84-b89f-449ce5ae621b" containerName="payload-pre-processing" containerID="cri-o://a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b" gracePeriod=30
Oct 30 13:11:18.766170 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.766130    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"]
Oct 30 13:11:18.769433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.769409    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ffbba3-7c9cfb4d54-9zq7f"]
Oct 30 13:11:18.842390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.842367    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:11:18.979472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.978768    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f8256f-4dee-4d1a-91e6-8cfd197c1362-kube-api-access-c7fnq\" (UniqueName: \"kubernetes.io/projected/97f8256f-4dee-4d1a-91e6-8cfd197c1362-kube-api-access-c7fnq\") pod \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") "
Oct 30 13:11:18.979472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.978876    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f8256f-4dee-4d1a-91e6-8cfd197c1362-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f8256f-4dee-4d1a-91e6-8cfd197c1362-openshift-service-ca\") pod \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") "
Oct 30 13:11:18.979472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.978929    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-maas-api-tls\") pod \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") "
Oct 30 13:11:18.979472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.979065    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-metrics-certs\") pod \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\" (UID: \"97f8256f-4dee-4d1a-91e6-8cfd197c1362\") "
Oct 30 13:11:18.979971 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.979929    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f8256f-4dee-4d1a-91e6-8cfd197c1362-openshift-service-ca" pod "97f8256f-4dee-4d1a-91e6-8cfd197c1362" (UID: "97f8256f-4dee-4d1a-91e6-8cfd197c1362"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:18.983065 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.983013    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f8256f-4dee-4d1a-91e6-8cfd197c1362-kube-api-access-c7fnq" pod "97f8256f-4dee-4d1a-91e6-8cfd197c1362" (UID: "97f8256f-4dee-4d1a-91e6-8cfd197c1362"). InnerVolumeSpecName "kube-api-access-c7fnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:18.984034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.984008    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-maas-api-tls" pod "97f8256f-4dee-4d1a-91e6-8cfd197c1362" (UID: "97f8256f-4dee-4d1a-91e6-8cfd197c1362"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:18.984472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:18.984446    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-metrics-certs" pod "97f8256f-4dee-4d1a-91e6-8cfd197c1362" (UID: "97f8256f-4dee-4d1a-91e6-8cfd197c1362"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:19.088788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.088416    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f8256f-4dee-4d1a-91e6-8cfd197c1362-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:19.088788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.088451    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:19.088788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.088467    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f8256f-4dee-4d1a-91e6-8cfd197c1362-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:19.088788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.088481    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7fnq\" (UniqueName: \"kubernetes.io/projected/97f8256f-4dee-4d1a-91e6-8cfd197c1362-kube-api-access-c7fnq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:19.139927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.139901    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:11:19.189314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.188949    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188768bd-7283-4c84-b89f-449ce5ae621b-kube-api-access-6nrds\" (UniqueName: \"kubernetes.io/projected/188768bd-7283-4c84-b89f-449ce5ae621b-kube-api-access-6nrds\") pod \"188768bd-7283-4c84-b89f-449ce5ae621b\" (UID: \"188768bd-7283-4c84-b89f-449ce5ae621b\") "
Oct 30 13:11:19.189314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.189258    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188768bd-7283-4c84-b89f-449ce5ae621b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188768bd-7283-4c84-b89f-449ce5ae621b-plugins-config-volume\") pod \"188768bd-7283-4c84-b89f-449ce5ae621b\" (UID: \"188768bd-7283-4c84-b89f-449ce5ae621b\") "
Oct 30 13:11:19.190064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.189991    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188768bd-7283-4c84-b89f-449ce5ae621b-plugins-config-volume" pod "188768bd-7283-4c84-b89f-449ce5ae621b" (UID: "188768bd-7283-4c84-b89f-449ce5ae621b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:19.199123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.199057    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188768bd-7283-4c84-b89f-449ce5ae621b-kube-api-access-6nrds" pod "188768bd-7283-4c84-b89f-449ce5ae621b" (UID: "188768bd-7283-4c84-b89f-449ce5ae621b"). InnerVolumeSpecName "kube-api-access-6nrds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:19.291215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.291186    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/188768bd-7283-4c84-b89f-449ce5ae621b-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:19.291215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.291213    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrds\" (UniqueName: \"kubernetes.io/projected/188768bd-7283-4c84-b89f-449ce5ae621b-kube-api-access-6nrds\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:19.693812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.693777    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"
Oct 30 13:11:19.693970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.693802    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8" event={"ID":"97f8256f-4dee-4d1a-91e6-8cfd197c1362","Type":"ContainerDied","Data":"bb306f5400d1b5097112ca001a84e0c785e937b655a008044ad1e4329bff00ab"}
Oct 30 13:11:19.693970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.693876    4429 scope.go:122] "RemoveContainer" containerID="eda4f335c56e792023d4a62203c60c83ad12881acbfab330924cd40ccdbcee57"
Oct 30 13:11:19.696711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.696669    4429 generic.go:356] "Generic (PLEG): container finished" podID="188768bd-7283-4c84-b89f-449ce5ae621b" containerID="a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b" exitCode=0
Oct 30 13:11:19.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.696724    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"
Oct 30 13:11:19.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.696796    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2" event={"ID":"188768bd-7283-4c84-b89f-449ce5ae621b","Type":"ContainerDied","Data":"a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b"}
Oct 30 13:11:19.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.696829    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2" event={"ID":"188768bd-7283-4c84-b89f-449ce5ae621b","Type":"ContainerDied","Data":"acdb5cdecb2cc31198def3c491ae8b05c9a124ab90784748695de75a6d37fb0b"}
Oct 30 13:11:19.707541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.707439    4429 scope.go:122] "RemoveContainer" containerID="a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b"
Oct 30 13:11:19.720578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.720551    4429 scope.go:122] "RemoveContainer" containerID="a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b"
Oct 30 13:11:19.720875 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:11:19.720852    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b\": container with ID starting with a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b not found: ID does not exist" containerID="a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b"
Oct 30 13:11:19.720951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.720885    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b"} err="failed to get container status \"a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b\": rpc error: code = NotFound desc = could not find container \"a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b\": container with ID starting with a0ea685e03415f429b4e7f13800855dd9831be7c7c2bd49bfa72731375a8624b not found: ID does not exist"
Oct 30 13:11:19.814371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.814341    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"]
Oct 30 13:11:19.845385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.845360    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76c107-88854f498-v9hx8"]
Oct 30 13:11:19.897687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.897649    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"]
Oct 30 13:11:19.898301 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.898261    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-76c107-59f7896687-s2tf2"]
Oct 30 13:11:19.931352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.931313    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq" podUID="aaeba70d-8acf-4150-83e5-9f95d6832a1a" containerName="payload-processing" containerID="cri-o://5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8" gracePeriod=25
Oct 30 13:11:19.981151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:19.981096    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58" podUID="beaf5b45-651c-4ccc-89a2-70a03290b1d7" containerName="payload-pre-processing" containerID="cri-o://3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700" gracePeriod=25
Oct 30 13:11:20.075731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.075698    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188768bd-7283-4c84-b89f-449ce5ae621b" path="/var/lib/kubelet/pods/188768bd-7283-4c84-b89f-449ce5ae621b/volumes"
Oct 30 13:11:20.076275 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.076252    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f8256f-4dee-4d1a-91e6-8cfd197c1362" path="/var/lib/kubelet/pods/97f8256f-4dee-4d1a-91e6-8cfd197c1362/volumes"
Oct 30 13:11:20.076798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.076777    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d264e49b-9129-42c0-98ee-6bcf79461e8b" path="/var/lib/kubelet/pods/d264e49b-9129-42c0-98ee-6bcf79461e8b/volumes"
Oct 30 13:11:20.253032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.253008    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:11:20.303722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.303659    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaeba70d-8acf-4150-83e5-9f95d6832a1a-kube-api-access-zr5nc\" (UniqueName: \"kubernetes.io/projected/aaeba70d-8acf-4150-83e5-9f95d6832a1a-kube-api-access-zr5nc\") pod \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\" (UID: \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\") "
Oct 30 13:11:20.303832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.303735    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaeba70d-8acf-4150-83e5-9f95d6832a1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaeba70d-8acf-4150-83e5-9f95d6832a1a-plugins-config-volume\") pod \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\" (UID: \"aaeba70d-8acf-4150-83e5-9f95d6832a1a\") "
Oct 30 13:11:20.304130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.304107    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaeba70d-8acf-4150-83e5-9f95d6832a1a-plugins-config-volume" pod "aaeba70d-8acf-4150-83e5-9f95d6832a1a" (UID: "aaeba70d-8acf-4150-83e5-9f95d6832a1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:20.304442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.304426    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaeba70d-8acf-4150-83e5-9f95d6832a1a-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:20.306231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.306203    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaeba70d-8acf-4150-83e5-9f95d6832a1a-kube-api-access-zr5nc" pod "aaeba70d-8acf-4150-83e5-9f95d6832a1a" (UID: "aaeba70d-8acf-4150-83e5-9f95d6832a1a"). InnerVolumeSpecName "kube-api-access-zr5nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:20.324803 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.324779    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:11:20.405711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.405678    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beaf5b45-651c-4ccc-89a2-70a03290b1d7-kube-api-access-84n9s\" (UniqueName: \"kubernetes.io/projected/beaf5b45-651c-4ccc-89a2-70a03290b1d7-kube-api-access-84n9s\") pod \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\" (UID: \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\") "
Oct 30 13:11:20.405880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.405782    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beaf5b45-651c-4ccc-89a2-70a03290b1d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beaf5b45-651c-4ccc-89a2-70a03290b1d7-plugins-config-volume\") pod \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\" (UID: \"beaf5b45-651c-4ccc-89a2-70a03290b1d7\") "
Oct 30 13:11:20.406233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.406206    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beaf5b45-651c-4ccc-89a2-70a03290b1d7-plugins-config-volume" pod "beaf5b45-651c-4ccc-89a2-70a03290b1d7" (UID: "beaf5b45-651c-4ccc-89a2-70a03290b1d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:20.406464 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.406438    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr5nc\" (UniqueName: \"kubernetes.io/projected/aaeba70d-8acf-4150-83e5-9f95d6832a1a-kube-api-access-zr5nc\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:20.406598 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.406467    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beaf5b45-651c-4ccc-89a2-70a03290b1d7-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:20.408078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.408054    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beaf5b45-651c-4ccc-89a2-70a03290b1d7-kube-api-access-84n9s" pod "beaf5b45-651c-4ccc-89a2-70a03290b1d7" (UID: "beaf5b45-651c-4ccc-89a2-70a03290b1d7"). InnerVolumeSpecName "kube-api-access-84n9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:20.508117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.508050    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84n9s\" (UniqueName: \"kubernetes.io/projected/beaf5b45-651c-4ccc-89a2-70a03290b1d7-kube-api-access-84n9s\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:20.704971 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.704940    4429 generic.go:356] "Generic (PLEG): container finished" podID="aaeba70d-8acf-4150-83e5-9f95d6832a1a" containerID="5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8" exitCode=0
Oct 30 13:11:20.705128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.704987    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"
Oct 30 13:11:20.705128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.705022    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq" event={"ID":"aaeba70d-8acf-4150-83e5-9f95d6832a1a","Type":"ContainerDied","Data":"5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8"}
Oct 30 13:11:20.705128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.705082    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq" event={"ID":"aaeba70d-8acf-4150-83e5-9f95d6832a1a","Type":"ContainerDied","Data":"fcc1fdd9a23c0aae63b31af9e372bf524e58139d2032929752cc1c2a767282bc"}
Oct 30 13:11:20.705128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.705117    4429 scope.go:122] "RemoveContainer" containerID="5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8"
Oct 30 13:11:20.706368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.706342    4429 generic.go:356] "Generic (PLEG): container finished" podID="beaf5b45-651c-4ccc-89a2-70a03290b1d7" containerID="3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700" exitCode=0
Oct 30 13:11:20.706505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.706375    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58" event={"ID":"beaf5b45-651c-4ccc-89a2-70a03290b1d7","Type":"ContainerDied","Data":"3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700"}
Oct 30 13:11:20.706505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.706387    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"
Oct 30 13:11:20.706505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.706417    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58" event={"ID":"beaf5b45-651c-4ccc-89a2-70a03290b1d7","Type":"ContainerDied","Data":"49c3016a0c10868cab86543c95ddff84d91170d856fa62d0a24d02c6bec405ec"}
Oct 30 13:11:20.720491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.720470    4429 scope.go:122] "RemoveContainer" containerID="5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8"
Oct 30 13:11:20.720918 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:11:20.720778    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8\": container with ID starting with 5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8 not found: ID does not exist" containerID="5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8"
Oct 30 13:11:20.720918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.720809    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8"} err="failed to get container status \"5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8\": rpc error: code = NotFound desc = could not find container \"5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8\": container with ID starting with 5b99f74b6385deb132f85f38878162e88805ee721208e9be3d7c2e930ee0c5c8 not found: ID does not exist"
Oct 30 13:11:20.720918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.720829    4429 scope.go:122] "RemoveContainer" containerID="3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700"
Oct 30 13:11:20.730764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.730748    4429 scope.go:122] "RemoveContainer" containerID="3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700"
Oct 30 13:11:20.731024 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:11:20.731006    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700\": container with ID starting with 3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700 not found: ID does not exist" containerID="3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700"
Oct 30 13:11:20.731073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.731032    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700"} err="failed to get container status \"3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700\": rpc error: code = NotFound desc = could not find container \"3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700\": container with ID starting with 3dc5ed88dbc680747ed7d8017f40b8d8cc214e9c83e6986f9bceba2370fbe700 not found: ID does not exist"
Oct 30 13:11:20.761745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.761691    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"]
Oct 30 13:11:20.775238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.775212    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ffbba3-56d7db84c5-rhb58"]
Oct 30 13:11:20.812342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.812318    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"]
Oct 30 13:11:20.815763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:20.815744    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ffbba3-79b8659d9d-wjmvq"]
Oct 30 13:11:22.076944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:22.076906    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaeba70d-8acf-4150-83e5-9f95d6832a1a" path="/var/lib/kubelet/pods/aaeba70d-8acf-4150-83e5-9f95d6832a1a/volumes"
Oct 30 13:11:22.077393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:22.077378    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beaf5b45-651c-4ccc-89a2-70a03290b1d7" path="/var/lib/kubelet/pods/beaf5b45-651c-4ccc-89a2-70a03290b1d7/volumes"
Oct 30 13:11:23.146737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:23.146700    4429 scope.go:122] "RemoveContainer" containerID="fc87680d6783c806d0c7a6ceca2751965edcda83e0dc4b2b3645b6991fac2721"
Oct 30 13:11:26.646007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.645977    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"]
Oct 30 13:11:26.646806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646786    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d264e49b-9129-42c0-98ee-6bcf79461e8b" containerName="maas-api"
Oct 30 13:11:26.646927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646808    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d264e49b-9129-42c0-98ee-6bcf79461e8b" containerName="maas-api"
Oct 30 13:11:26.646927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646830    4429 cpu_manager.go:414] "containerMap: removing container" podUID="188768bd-7283-4c84-b89f-449ce5ae621b" containerName="payload-pre-processing"
Oct 30 13:11:26.646927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646838    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188768bd-7283-4c84-b89f-449ce5ae621b" containerName="payload-pre-processing"
Oct 30 13:11:26.646927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646877    4429 cpu_manager.go:414] "containerMap: removing container" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy"
Oct 30 13:11:26.646927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646885    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy"
Oct 30 13:11:26.646927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646901    4429 cpu_manager.go:414] "containerMap: removing container" podUID="beaf5b45-651c-4ccc-89a2-70a03290b1d7" containerName="payload-pre-processing"
Oct 30 13:11:26.646927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646907    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beaf5b45-651c-4ccc-89a2-70a03290b1d7" containerName="payload-pre-processing"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646935    4429 cpu_manager.go:414] "containerMap: removing container" podUID="97f8256f-4dee-4d1a-91e6-8cfd197c1362" containerName="maas-api"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646942    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f8256f-4dee-4d1a-91e6-8cfd197c1362" containerName="maas-api"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646957    4429 cpu_manager.go:414] "containerMap: removing container" podUID="aaeba70d-8acf-4150-83e5-9f95d6832a1a" containerName="payload-processing"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.646964    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaeba70d-8acf-4150-83e5-9f95d6832a1a" containerName="payload-processing"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.647095    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f8256f-4dee-4d1a-91e6-8cfd197c1362" containerName="maas-api"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.647111    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaeba70d-8acf-4150-83e5-9f95d6832a1a" containerName="payload-processing"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.647122    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d264e49b-9129-42c0-98ee-6bcf79461e8b" containerName="maas-api"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.647130    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="188768bd-7283-4c84-b89f-449ce5ae621b" containerName="payload-pre-processing"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.647146    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbd3ed2-7a4b-4191-bc84-67042b85dc8b" containerName="istio-proxy"
Oct 30 13:11:26.647324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.647156    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="beaf5b45-651c-4ccc-89a2-70a03290b1d7" containerName="payload-pre-processing"
Oct 30 13:11:26.652569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.652548    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.693951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.691967    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"]
Oct 30 13:11:26.696380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.696355    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"]
Oct 30 13:11:26.696497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.696476    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.720547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.717509    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"]
Oct 30 13:11:26.792728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.792694    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.792728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.792732    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.792929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.792812    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.792929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.792840    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.792999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.792924    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.792999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.792971    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.793067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793017    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.793067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793049    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjt98\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-kube-api-access-mjt98\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.793127 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793070    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab95c54-be84-4760-ab71-1d017403a629-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.793161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793151    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.793195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793181    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.793342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793327    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab95c54-be84-4760-ab71-1d017403a629-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.793392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793363    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.793429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793390    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.793464 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793433    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.793494 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793465    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.793527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793501    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.793561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.793534    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6rv\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-kube-api-access-kn6rv\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.885686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.885656    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"]
Oct 30 13:11:26.885922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.885885    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy" containerID="cri-o://ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef" gracePeriod=30
Oct 30 13:11:26.894952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.894925    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.894972    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.895070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895001    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895032    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjt98\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-kube-api-access-mjt98\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895061    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab95c54-be84-4760-ab71-1d017403a629-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895108    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895138    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895255    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab95c54-be84-4760-ab71-1d017403a629-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895420    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.895482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895429    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895484    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895396    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.895593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895564    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.895767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895609    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.895767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895564    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895663    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.895767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895698    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.895767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895736    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6rv\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-kube-api-access-kn6rv\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.896054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895786    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.896054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895815    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.896054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895854    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab95c54-be84-4760-ab71-1d017403a629-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.896054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895883    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.896054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.895916    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.896470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.896444    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.896558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.896483    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-data\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.896783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.896759    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-certs\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.896858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.896763    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-credential-socket\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.897061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.897040    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istiod-ca-cert\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.897964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.897946    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.898081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.898064    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab95c54-be84-4760-ab71-1d017403a629-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.898142 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.898122    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-envoy\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.898400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.898382    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-podinfo\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.931390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.931356    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjt98\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-kube-api-access-mjt98\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.932612 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.932582    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.932848 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.932825    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6rv\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-kube-api-access-kn6rv\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:26.935856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.935827    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-istio-token\") pod \"e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") " pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:26.967484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:26.967459    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:27.035312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.024958    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:27.392056 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:11:27.392028    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab95c54_be84_4760_ab71_1d017403a629.slice/crio-8d23007edd405abccc122c618beeab2b0ef329266c214f9542e88ae7ca3abc25 WatchSource:0}: Error finding container 8d23007edd405abccc122c618beeab2b0ef329266c214f9542e88ae7ca3abc25: Status 404 returned error can't find the container with id 8d23007edd405abccc122c618beeab2b0ef329266c214f9542e88ae7ca3abc25
Oct 30 13:11:27.393107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.393084    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"]
Oct 30 13:11:27.394691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.394662    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:27.394742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.394725    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:27.394777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.394752    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:27.396018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.395992    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"]
Oct 30 13:11:27.403853 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:11:27.403829    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7860b3b_699d_424b_8a3e_ff32cf1a6cda.slice/crio-17ed08bbc625c83e184b442ec96aca1d779af9375c97613d43d0427140899690 WatchSource:0}: Error finding container 17ed08bbc625c83e184b442ec96aca1d779af9375c97613d43d0427140899690: Status 404 returned error can't find the container with id 17ed08bbc625c83e184b442ec96aca1d779af9375c97613d43d0427140899690
Oct 30 13:11:27.406489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.406444    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:27.406558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.406505    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:27.406558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.406533    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:27.755503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.755465    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" event={"ID":"cab95c54-be84-4760-ab71-1d017403a629","Type":"ContainerStarted","Data":"79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f"}
Oct 30 13:11:27.756033 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.755524    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" event={"ID":"cab95c54-be84-4760-ab71-1d017403a629","Type":"ContainerStarted","Data":"8d23007edd405abccc122c618beeab2b0ef329266c214f9542e88ae7ca3abc25"}
Oct 30 13:11:27.757515 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.757487    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" event={"ID":"f7860b3b-699d-424b-8a3e-ff32cf1a6cda","Type":"ContainerStarted","Data":"9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca"}
Oct 30 13:11:27.757633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.757544    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" event={"ID":"f7860b3b-699d-424b-8a3e-ff32cf1a6cda","Type":"ContainerStarted","Data":"17ed08bbc625c83e184b442ec96aca1d779af9375c97613d43d0427140899690"}
Oct 30 13:11:27.807028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.806972    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podStartSLOduration=1.806955022 podStartE2EDuration="1.806955022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:27.803599021 +0000 UTC m=+2345.859092868" watchObservedRunningTime="2026-10-30 13:11:27.806955022 +0000 UTC m=+2345.862448847"
Oct 30 13:11:27.855245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.855194    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podStartSLOduration=1.8551803059999998 podStartE2EDuration="1.855180306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:27.851372118 +0000 UTC m=+2345.906865956" watchObservedRunningTime="2026-10-30 13:11:27.855180306 +0000 UTC m=+2345.910674130"
Oct 30 13:11:27.967948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:27.967921    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:28.027180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:28.027102    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:28.029052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:28.029017    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:28.029187 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:28.029062    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:28.993309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:28.989813    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:11:28.993309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:28.989890    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 13:11:29.027754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:29.027716    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:29.027929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:29.027781    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:29.990310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:29.987651    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:11:29.990310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:29.987710    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 13:11:30.027407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:30.027377    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:30.027771 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:30.027436    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:30.978306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:30.978220    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:11:30.999308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:30.978725    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 13:11:31.027402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:31.027369    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:31.027905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:31.027429    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:31.981009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:31.980394    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:11:31.981009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:31.980463    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 13:11:32.028200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.028167    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:32.028685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.028243    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:32.428419 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.428389    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:11:32.531877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.531842    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-podinfo\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.531928    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-token\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-token\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.531949    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istiod-ca-cert\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.531980    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-kube-api-access-z594h\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-kube-api-access-z594h\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532000    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-credential-socket\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532020    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-certs\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532092    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-socket\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532163    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-data\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532188    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-envoy\") pod \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\" (UID: \"1a805ee4-c7f9-4e63-a427-ec946740e9bf\") "
Oct 30 13:11:32.532683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532653    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-certs" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.532836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532731    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istiod-ca-cert" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:32.532836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532812    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-socket" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.532946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532849    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-data" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.532946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.532915    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-credential-socket" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.534130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.534108    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-podinfo" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:32.534855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.534830    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-kube-api-access-z594h" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "kube-api-access-z594h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.535185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.535163    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-envoy" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.535312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.535237    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-token" pod "1a805ee4-c7f9-4e63-a427-ec946740e9bf" (UID: "1a805ee4-c7f9-4e63-a427-ec946740e9bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.634102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634023    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z594h\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-kube-api-access-z594h\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.634102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634058    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.634102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634073    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.634102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634090    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.634354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634105    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.634354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634121    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.634354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634133    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.634354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634144    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.634354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.634158    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a805ee4-c7f9-4e63-a427-ec946740e9bf-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.836050 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.836017    4429 generic.go:356] "Generic (PLEG): container finished" podID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerID="ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef" exitCode=0
Oct 30 13:11:32.836226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.836065    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"
Oct 30 13:11:32.836226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.836099    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" event={"ID":"1a805ee4-c7f9-4e63-a427-ec946740e9bf","Type":"ContainerDied","Data":"ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef"}
Oct 30 13:11:32.836226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.836168    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng" event={"ID":"1a805ee4-c7f9-4e63-a427-ec946740e9bf","Type":"ContainerDied","Data":"503d2f7aa5b881e881d9c5946ae17db31f2070e5da9a896a600d2e0126a14522"}
Oct 30 13:11:32.836226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.836193    4429 scope.go:122] "RemoveContainer" containerID="ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef"
Oct 30 13:11:32.846569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.846552    4429 scope.go:122] "RemoveContainer" containerID="c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794"
Oct 30 13:11:32.907499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.907472    4429 scope.go:122] "RemoveContainer" containerID="ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef"
Oct 30 13:11:32.907930 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:11:32.907908    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef\": container with ID starting with ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef not found: ID does not exist" containerID="ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef"
Oct 30 13:11:32.908066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.908046    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef"} err="failed to get container status \"ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef\": rpc error: code = NotFound desc = could not find container \"ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef\": container with ID starting with ab8f5701f9edf683a8dbef4ae7889e447b680579762b6df2abf5df0c7f6b52ef not found: ID does not exist"
Oct 30 13:11:32.908136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.908129    4429 scope.go:122] "RemoveContainer" containerID="c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794"
Oct 30 13:11:32.908428 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:11:32.908414    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794\": container with ID starting with c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794 not found: ID does not exist" containerID="c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794"
Oct 30 13:11:32.908519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.908506    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794"} err="failed to get container status \"c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794\": rpc error: code = NotFound desc = could not find container \"c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794\": container with ID starting with c43ec2e66a2aaca2924422582540e1788fc556d035f171e8d839602b4b749794 not found: ID does not exist"
Oct 30 13:11:32.923666 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.923645    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"]
Oct 30 13:11:32.936337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.936315    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ffbba3-openshift-default-6978cc47b5-7kwng"]
Oct 30 13:11:32.982309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.981623    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:11:32.982309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:32.981697    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 13:11:33.027675 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:33.027639    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:33.027835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:33.027681    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:33.972310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:33.971730    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:11:33.972310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:33.971796    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 13:11:34.044312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:34.030692    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:34.044312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:34.030754    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:34.112310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:34.111390    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" path="/var/lib/kubelet/pods/1a805ee4-c7f9-4e63-a427-ec946740e9bf/volumes"
Oct 30 13:11:34.975311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:34.975248    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 13:11:34.975877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:34.975347    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 13:11:35.051310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:35.028898    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:35.051310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:35.028972    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:35.384701 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:35.376533    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:11:35.421639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:35.420409    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.036311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.030403    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 13:11:36.036311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.030462    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 13:11:36.179312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.157900    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:36.179312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.157978    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.231310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.216425    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.383364 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.382903    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.383364 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.383017    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.408309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.392932    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:36.408309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.394438    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:11:36.444307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.443633    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 13:11:36.444307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:36.444169    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:11:37.253328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.221195    4429 patch_prober.go:29] interesting pod/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:37.253328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.221256    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.253328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.223103    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.253328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.223456    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:11:37.408303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.396273    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.408303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.397236    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.408303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.397734    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:11:37.408303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.405778    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:37.408303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.406226    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:11:37.470983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.470957    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:11:37.493955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.493928    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:11:37.494542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.494460    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:11:37.525424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.511748    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:11:37.525424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.519941    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:11:37.615824 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.615783    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:37.617613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.617590    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:37.619981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.619961    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:11:37.642024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.641971    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:37.643356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.643327    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:37.645155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:37.645121    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:11:38.397309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.367045    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"]
Oct 30 13:11:38.397309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.367988    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy"
Oct 30 13:11:38.397309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.368009    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy"
Oct 30 13:11:38.397309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.368032    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy"
Oct 30 13:11:38.397309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.368040    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy"
Oct 30 13:11:38.397309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.368170    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy"
Oct 30 13:11:38.397309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.368183    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a805ee4-c7f9-4e63-a427-ec946740e9bf" containerName="istio-proxy"
Oct 30 13:11:38.397309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.396910    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:38.418755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.417546    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-2fdf91\""
Oct 30 13:11:38.418755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.418254    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-2fdf91-dockercfg-wxqml\""
Oct 30 13:11:38.426463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.426442    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"]
Oct 30 13:11:38.438602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.438497    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj\" (UID: \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:38.438602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.438565    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsw8n\" (UniqueName: \"kubernetes.io/projected/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-kube-api-access-wsw8n\") pod \"payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj\" (UID: \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:38.691919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.572620    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj\" (UID: \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:38.691919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.572683    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsw8n\" (UniqueName: \"kubernetes.io/projected/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-kube-api-access-wsw8n\") pod \"payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj\" (UID: \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:38.691919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.574008    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj\" (UID: \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:38.719752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.719710    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"]
Oct 30 13:11:38.760549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.760515    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsw8n\" (UniqueName: \"kubernetes.io/projected/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-kube-api-access-wsw8n\") pod \"payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj\" (UID: \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:38.773391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.762147    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"]
Oct 30 13:11:38.820308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.762694    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:38.820308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.791152    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa257796-b27a-4909-abd7-d99506b2addb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx\" (UID: \"aa257796-b27a-4909-abd7-d99506b2addb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:38.822369 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.820588    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k55r\" (UniqueName: \"kubernetes.io/projected/aa257796-b27a-4909-abd7-d99506b2addb-kube-api-access-9k55r\") pod \"payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx\" (UID: \"aa257796-b27a-4909-abd7-d99506b2addb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:38.921762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.921717    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa257796-b27a-4909-abd7-d99506b2addb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx\" (UID: \"aa257796-b27a-4909-abd7-d99506b2addb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:38.922029 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.922010    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k55r\" (UniqueName: \"kubernetes.io/projected/aa257796-b27a-4909-abd7-d99506b2addb-kube-api-access-9k55r\") pod \"payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx\" (UID: \"aa257796-b27a-4909-abd7-d99506b2addb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:38.922450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.922428    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa257796-b27a-4909-abd7-d99506b2addb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx\" (UID: \"aa257796-b27a-4909-abd7-d99506b2addb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:38.977351 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:38.977202    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k55r\" (UniqueName: \"kubernetes.io/projected/aa257796-b27a-4909-abd7-d99506b2addb-kube-api-access-9k55r\") pod \"payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx\" (UID: \"aa257796-b27a-4909-abd7-d99506b2addb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:39.027936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:39.027907    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:39.169317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:39.168627    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:39.407061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:39.407030    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"]
Oct 30 13:11:39.407469 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:11:39.407078    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec8dc83_4eb9_4c1a_a724_df8d7438a0fc.slice/crio-8fe04bd13d9ea7f38a7eb9e41fd2e3d73872c58c0b8d137ed5d1b5df5cf60dc2 WatchSource:0}: Error finding container 8fe04bd13d9ea7f38a7eb9e41fd2e3d73872c58c0b8d137ed5d1b5df5cf60dc2: Status 404 returned error can't find the container with id 8fe04bd13d9ea7f38a7eb9e41fd2e3d73872c58c0b8d137ed5d1b5df5cf60dc2
Oct 30 13:11:39.547401 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:11:39.547371    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa257796_b27a_4909_abd7_d99506b2addb.slice/crio-ee06cfe10e6a113f281bb87761b55e7f5924a63ed6295a309453bc62f8f18e8c WatchSource:0}: Error finding container ee06cfe10e6a113f281bb87761b55e7f5924a63ed6295a309453bc62f8f18e8c: Status 404 returned error can't find the container with id ee06cfe10e6a113f281bb87761b55e7f5924a63ed6295a309453bc62f8f18e8c
Oct 30 13:11:39.547754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:39.547732    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"]
Oct 30 13:11:40.337098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:40.337059    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx" event={"ID":"aa257796-b27a-4909-abd7-d99506b2addb","Type":"ContainerStarted","Data":"d0656018a3b5266597511083a5e7fe238721ce6e8006406b7369fee83992e2f3"}
Oct 30 13:11:40.337359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:40.337141    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:40.337359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:40.337167    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx" event={"ID":"aa257796-b27a-4909-abd7-d99506b2addb","Type":"ContainerStarted","Data":"ee06cfe10e6a113f281bb87761b55e7f5924a63ed6295a309453bc62f8f18e8c"}
Oct 30 13:11:40.339120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:40.339083    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj" event={"ID":"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc","Type":"ContainerStarted","Data":"953d6dbbd4e998752d37eb8a6c16999ebf8e5669fe55b73f74ddf5d2aae55630"}
Oct 30 13:11:40.339259 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:40.339145    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:40.339259 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:40.339169    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj" event={"ID":"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc","Type":"ContainerStarted","Data":"8fe04bd13d9ea7f38a7eb9e41fd2e3d73872c58c0b8d137ed5d1b5df5cf60dc2"}
Oct 30 13:11:40.385904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:40.385852    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx" podStartSLOduration=2.385838574 podStartE2EDuration="2.385838574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:40.382956475 +0000 UTC m=+2358.438450319" watchObservedRunningTime="2026-10-30 13:11:40.385838574 +0000 UTC m=+2358.441332398"
Oct 30 13:11:40.439634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:40.439583    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj" podStartSLOduration=2.439570255 podStartE2EDuration="2.439570255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:40.437186579 +0000 UTC m=+2358.492680403" watchObservedRunningTime="2026-10-30 13:11:40.439570255 +0000 UTC m=+2358.495064122"
Oct 30 13:11:42.373609 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:11:42.373567    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:11:45.546484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.546445    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"]
Oct 30 13:11:45.551873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.551848    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.566125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.566099    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"]
Oct 30 13:11:45.576228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.576045    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e9cb4a-openshift-default-dockercfg-5lrtb\""
Oct 30 13:11:45.619063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619039    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-certs\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.619157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619076    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-envoy\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.619157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619115    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.619263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619245    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-token\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.619361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619346    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwdg7\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-kube-api-access-wwdg7\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.619442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619420    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d2e60-03b5-4ae7-b439-3b57d47d2902-istiod-ca-cert\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.619475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619459    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-data\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.619513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619499    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-podinfo\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.619566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.619555    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-credential-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.721487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.721450    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-certs\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.721776 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.721734    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-envoy\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.721958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.721819    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.721958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.721905    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-token\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.722103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.721976    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwdg7\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-kube-api-access-wwdg7\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.722103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.722057    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d2e60-03b5-4ae7-b439-3b57d47d2902-istiod-ca-cert\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.722211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.722115    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-data\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.722211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.722141    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-podinfo\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.722211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.722203    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-credential-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.722743 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.722718    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-credential-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.728654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.722991    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-certs\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.728843 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.723975    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-socket\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.728952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.724438    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-data\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.729051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.725017    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d2e60-03b5-4ae7-b439-3b57d47d2902-istiod-ca-cert\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.740855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.740792    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-envoy\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.743580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.743534    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-podinfo\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.765070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.765004    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwdg7\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-kube-api-access-wwdg7\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.765858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.765807    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-token\") pod \"e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") " pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:45.868812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:45.868717    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:46.208073 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:11:46.208041    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922d2e60_03b5_4ae7_b439_3b57d47d2902.slice/crio-b78c221fa995fb06a0823041fac04581e12c6948b64ae10dec623863ad7d81ea WatchSource:0}: Error finding container b78c221fa995fb06a0823041fac04581e12c6948b64ae10dec623863ad7d81ea: Status 404 returned error can't find the container with id b78c221fa995fb06a0823041fac04581e12c6948b64ae10dec623863ad7d81ea
Oct 30 13:11:46.208823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:46.208801    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"]
Oct 30 13:11:46.210636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:46.210609    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:46.210706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:46.210672    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:46.210749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:46.210719    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:46.377724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:46.377691    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" event={"ID":"922d2e60-03b5-4ae7-b439-3b57d47d2902","Type":"ContainerStarted","Data":"eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f"}
Oct 30 13:11:46.377859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:46.377760    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" event={"ID":"922d2e60-03b5-4ae7-b439-3b57d47d2902","Type":"ContainerStarted","Data":"b78c221fa995fb06a0823041fac04581e12c6948b64ae10dec623863ad7d81ea"}
Oct 30 13:11:46.428475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:46.428418    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" podStartSLOduration=1.428403836 podStartE2EDuration="1.428403836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:46.427071084 +0000 UTC m=+2364.482564910" watchObservedRunningTime="2026-10-30 13:11:46.428403836 +0000 UTC m=+2364.483897729"
Oct 30 13:11:46.869637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:46.869597    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:47.874641 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:47.874611    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:48.389974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:48.389942    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:48.390788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:48.390771    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:11:51.348711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.348684    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:11:51.349157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.348751    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:11:51.434395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.434360    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"]
Oct 30 13:11:51.439711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.439689    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.513782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.513750    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"]
Oct 30 13:11:51.517778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.517513    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-7a8b38-openshift-default-dockercfg-c8j64\""
Oct 30 13:11:51.535427 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.535398    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"]
Oct 30 13:11:51.540811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.540789    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:51.564348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.564312    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e9cb4a\""
Oct 30 13:11:51.576584 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.576560    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"]
Oct 30 13:11:51.603510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603244    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-data\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.603510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603332    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgc75\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-kube-api-access-wgc75\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.603510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603398    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-istio-token\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.603510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603435    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-socket\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.603510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603474    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-certs\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.603881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603552    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327a616-e506-492f-88b7-66e4143df16a-istio-podinfo\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.603881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603604    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327a616-e506-492f-88b7-66e4143df16a-istiod-ca-cert\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.603881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603690    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-envoy\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.603881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.603782    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-credential-socket\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.627968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.627942    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e9cb4a-dockercfg-dpjv6\""
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.718725    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-istio-token\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.718801    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-socket\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.718836    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-certs\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.718905    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327a616-e506-492f-88b7-66e4143df16a-istio-podinfo\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.718951    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327a616-e506-492f-88b7-66e4143df16a-istiod-ca-cert\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.719001    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz799\" (UniqueName: \"kubernetes.io/projected/97305157-a41d-465e-8e94-1aa9dc38a166-kube-api-access-jz799\") pod \"payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6\" (UID: \"97305157-a41d-465e-8e94-1aa9dc38a166\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.719058    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-envoy\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.719136    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-credential-socket\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.719168    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97305157-a41d-465e-8e94-1aa9dc38a166-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6\" (UID: \"97305157-a41d-465e-8e94-1aa9dc38a166\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.719250    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-data\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.720310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.719316    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgc75\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-kube-api-access-wgc75\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.721035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.720396    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-socket\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.721035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.720648    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-certs\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.750309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.742669    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-credential-socket\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.750309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.742967    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-data\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.750309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.743680    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327a616-e506-492f-88b7-66e4143df16a-istiod-ca-cert\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.750309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.745828    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327a616-e506-492f-88b7-66e4143df16a-istio-podinfo\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.780307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.772827    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-envoy\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.794224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.794176    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-istio-token\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.795319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.795276    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgc75\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-kube-api-access-wgc75\") pod \"e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") " pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:51.821304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.821248    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97305157-a41d-465e-8e94-1aa9dc38a166-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6\" (UID: \"97305157-a41d-465e-8e94-1aa9dc38a166\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:51.822376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.822354    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz799\" (UniqueName: \"kubernetes.io/projected/97305157-a41d-465e-8e94-1aa9dc38a166-kube-api-access-jz799\") pod \"payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6\" (UID: \"97305157-a41d-465e-8e94-1aa9dc38a166\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:51.822594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.822102    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97305157-a41d-465e-8e94-1aa9dc38a166-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6\" (UID: \"97305157-a41d-465e-8e94-1aa9dc38a166\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:51.857915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.857866    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz799\" (UniqueName: \"kubernetes.io/projected/97305157-a41d-465e-8e94-1aa9dc38a166-kube-api-access-jz799\") pod \"payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6\" (UID: \"97305157-a41d-465e-8e94-1aa9dc38a166\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:51.861757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:51.861733    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:52.059388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.059358    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:52.183038 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:11:52.183006    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97305157_a41d_465e_8e94_1aa9dc38a166.slice/crio-bad8cefccd74b7f32757cb3bdc29bef25bd3b6dc306e81506408e1359107848c WatchSource:0}: Error finding container bad8cefccd74b7f32757cb3bdc29bef25bd3b6dc306e81506408e1359107848c: Status 404 returned error can't find the container with id bad8cefccd74b7f32757cb3bdc29bef25bd3b6dc306e81506408e1359107848c
Oct 30 13:11:52.183469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.183448    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"]
Oct 30 13:11:52.378027 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:11:52.377991    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb327a616_e506_492f_88b7_66e4143df16a.slice/crio-453d07283ff71918020635bbdb7173ff8df6822b4defc30befd2d883c12c5887 WatchSource:0}: Error finding container 453d07283ff71918020635bbdb7173ff8df6822b4defc30befd2d883c12c5887: Status 404 returned error can't find the container with id 453d07283ff71918020635bbdb7173ff8df6822b4defc30befd2d883c12c5887
Oct 30 13:11:52.378823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.378801    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"]
Oct 30 13:11:52.380841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.380803    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:52.380939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.380886    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:52.381031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.380936    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:52.423718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.423690    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6" event={"ID":"97305157-a41d-465e-8e94-1aa9dc38a166","Type":"ContainerStarted","Data":"110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3"}
Oct 30 13:11:52.423883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.423866    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:11:52.423931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.423910    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6" event={"ID":"97305157-a41d-465e-8e94-1aa9dc38a166","Type":"ContainerStarted","Data":"bad8cefccd74b7f32757cb3bdc29bef25bd3b6dc306e81506408e1359107848c"}
Oct 30 13:11:52.428063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.427895    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" event={"ID":"b327a616-e506-492f-88b7-66e4143df16a","Type":"ContainerStarted","Data":"453d07283ff71918020635bbdb7173ff8df6822b4defc30befd2d883c12c5887"}
Oct 30 13:11:52.482215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:52.482153    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6" podStartSLOduration=1.4821346690000001 podStartE2EDuration="1.482134669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:52.479863913 +0000 UTC m=+2370.535357738" watchObservedRunningTime="2026-10-30 13:11:52.482134669 +0000 UTC m=+2370.537628496"
Oct 30 13:11:53.446068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:53.446032    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" event={"ID":"b327a616-e506-492f-88b7-66e4143df16a","Type":"ContainerStarted","Data":"63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6"}
Oct 30 13:11:53.521990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:53.521910    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" podStartSLOduration=2.521896923 podStartE2EDuration="2.521896923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:53.518979881 +0000 UTC m=+2371.574473705" watchObservedRunningTime="2026-10-30 13:11:53.521896923 +0000 UTC m=+2371.577390746"
Oct 30 13:11:54.060145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.060114    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:54.065792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.065767    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:54.452094 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.452066    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:54.453115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.453093    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:11:54.683798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.683769    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"]
Oct 30 13:11:54.683988 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.683971    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" containerID="cri-o://82e8d7aa3835a93308f4b670e847daca6053137d5a7c342ebfb4ed1b1d0d4c91" gracePeriod=30
Oct 30 13:11:54.688384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.688354    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"]
Oct 30 13:11:54.688619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.688595    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy" containerID="cri-o://40fffc686568d0dd8264e6c80807decd64715be0fe332256d692f81ad731d1db" gracePeriod=30
Oct 30 13:11:54.690101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.690081    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"]
Oct 30 13:11:54.690324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:54.690272    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy" containerID="cri-o://e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f" gracePeriod=30
Oct 30 13:11:58.502428 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:58.502378    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:58.540457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:58.540411    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:58.541108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:58.541078    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:11:58.581426 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:58.581386    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 13:11:58.883361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:58.882531    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Oct 30 13:11:59.401558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.401392    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.536448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.510619    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.578344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.514504    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.579095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.579072    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:11:59.580156 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.556764    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.581693 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.556798    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 13:11:59.774314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.591364    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:59.774314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.647044    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:11:59.774314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.724349    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:11:59.774314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.724385    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:11:59.854310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.849484    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-fks52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 13:11:59.854310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:11:59.849563    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 13:12:00.594312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.580922    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:12:00.620467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.596882    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 13:12:00.620467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.597608    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:12:00.620467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.598344    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 13:12:00.690234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.673100    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:12:00.699436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.697226    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 13:12:00.722276 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.721849    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:12:00.868013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.867520    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:12:00.879555 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.877942    4429 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:00.904084 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.903812    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-sjm4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 13:12:00.905380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.903866    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 13:12:00.912257 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.912202    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:12:00.920551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:00.920530    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:12:01.021438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.021408    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 13:12:01.062939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.062910    4429 generic.go:356] "Generic (PLEG): container finished" podID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerID="40fffc686568d0dd8264e6c80807decd64715be0fe332256d692f81ad731d1db" exitCode=0
Oct 30 13:12:01.063216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.063111    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" event={"ID":"3616efda-cc2c-4127-885e-49ea7d47fc12","Type":"ContainerDied","Data":"40fffc686568d0dd8264e6c80807decd64715be0fe332256d692f81ad731d1db"}
Oct 30 13:12:01.065196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.065156    4429 generic.go:356] "Generic (PLEG): container finished" podID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerID="e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f" exitCode=0
Oct 30 13:12:01.065339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.065204    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"
Oct 30 13:12:01.065339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.065228    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" event={"ID":"9a3f5772-16c2-4650-8bd8-fdb0ec149974","Type":"ContainerDied","Data":"e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f"}
Oct 30 13:12:01.065454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.065347    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" event={"ID":"9a3f5772-16c2-4650-8bd8-fdb0ec149974","Type":"ContainerDied","Data":"27be810bfab56f482e3e269bef2cd91644af7d565e3aa940c9b65b733fd76780"}
Oct 30 13:12:01.065454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.065406    4429 scope.go:122] "RemoveContainer" containerID="e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f"
Oct 30 13:12:01.067309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.067259    4429 generic.go:356] "Generic (PLEG): container finished" podID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerID="82e8d7aa3835a93308f4b670e847daca6053137d5a7c342ebfb4ed1b1d0d4c91" exitCode=0
Oct 30 13:12:01.067410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.067371    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" event={"ID":"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb","Type":"ContainerDied","Data":"82e8d7aa3835a93308f4b670e847daca6053137d5a7c342ebfb4ed1b1d0d4c91"}
Oct 30 13:12:01.067481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.067414    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52" event={"ID":"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb","Type":"ContainerDied","Data":"24816d24d3d6b8edd626b21ba33e8e93041ca6eca924fb43cc0a0f7e51bb8dae"}
Oct 30 13:12:01.067481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.067467    4429 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24816d24d3d6b8edd626b21ba33e8e93041ca6eca924fb43cc0a0f7e51bb8dae"
Oct 30 13:12:01.077137 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.077114    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 13:12:01.078191 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.078172    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-data\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.078471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.078437    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-data" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.078640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.078623    4429 scope.go:122] "RemoveContainer" containerID="e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f"
Oct 30 13:12:01.078970 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:12:01.078951    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f\": container with ID starting with e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f not found: ID does not exist" containerID="e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f"
Oct 30 13:12:01.079053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.078980    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f"} err="failed to get container status \"e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f\": rpc error: code = NotFound desc = could not find container \"e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f\": container with ID starting with e8cf242e82c9962cf267f2d7b29018be8ae68e9e3fa5f942c677970955e7df5f not found: ID does not exist"
Oct 30 13:12:01.079053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.079023    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.080656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.080636    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 13:12:01.180380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180346    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-token\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-token\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.180597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180405    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-data\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.180597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180435    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-credential-socket\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.180597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180468    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-kube-api-access-t7zr6\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-kube-api-access-t7zr6\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.180597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180549    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3616efda-cc2c-4127-885e-49ea7d47fc12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616efda-cc2c-4127-885e-49ea7d47fc12-istiod-ca-cert\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.180954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180698    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-data" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.180954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180712    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istiod-ca-cert\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.180954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180727    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-credential-socket" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.180954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180746    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-socket\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.180954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180789    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-certs\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.180954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180831    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-socket\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.180954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180871    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-envoy\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.180954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180926    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-podinfo\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.180969    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-certs\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181093    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-data\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181094    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3616efda-cc2c-4127-885e-49ea7d47fc12-istiod-ca-cert" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181123    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-kube-api-access-msxtw\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-kube-api-access-msxtw\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181158    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-envoy\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181195    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-certs\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181231    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-token\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-token\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181274    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-token\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-token\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181323    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-socket\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181379    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-credential-socket\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.181469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181422    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-kube-api-access-k9dbl\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-kube-api-access-k9dbl\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.182051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181478    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-podinfo\") pod \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\" (UID: \"b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb\") "
Oct 30 13:12:01.182051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181506    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-podinfo\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.182051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181597    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istiod-ca-cert\") pod \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\" (UID: \"9a3f5772-16c2-4650-8bd8-fdb0ec149974\") "
Oct 30 13:12:01.182051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181639    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-credential-socket\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.182051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181648    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-data" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.182051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.181949    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istiod-ca-cert" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:01.182470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182231    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-credential-socket" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.182525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182468    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.182525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182491    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.182525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182505    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3616efda-cc2c-4127-885e-49ea7d47fc12-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.182525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182517    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.182714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182522    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-token" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:01.182714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182528    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.182714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182560    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.182870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.182750    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-socket" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.183680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.183491    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-credential-socket" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.183680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.183495    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-socket" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.184135 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.183888    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-certs" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.184454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.184267    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-certs" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.184613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.184590    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-socket" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.185339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.185032    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istiod-ca-cert" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:01.185339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.185062    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-envoy" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.185339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.185264    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-kube-api-access-msxtw" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "kube-api-access-msxtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:01.185933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.185903    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-certs" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.186051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.186027    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-podinfo" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:01.186483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.186449    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-kube-api-access-t7zr6" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "kube-api-access-t7zr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:01.186836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.186815    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-podinfo" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:01.188711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.188683    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-kube-api-access-k9dbl" pod "9a3f5772-16c2-4650-8bd8-fdb0ec149974" (UID: "9a3f5772-16c2-4650-8bd8-fdb0ec149974"). InnerVolumeSpecName "kube-api-access-k9dbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:01.189146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.189126    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-token" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:01.189509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.189487    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-envoy" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.189509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.189501    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-podinfo" pod "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" (UID: "b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:01.191246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.191222    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-token" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:01.283660 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.283585    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-envoy\") pod \"3616efda-cc2c-4127-885e-49ea7d47fc12\" (UID: \"3616efda-cc2c-4127-885e-49ea7d47fc12\") "
Oct 30 13:12:01.284012 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.283995    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msxtw\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-kube-api-access-msxtw\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284015    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284024    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284033    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284041    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284051    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284063    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284072    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dbl\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-kube-api-access-k9dbl\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284080    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284087    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284095    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284107    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284118    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zr6\" (UniqueName: \"kubernetes.io/projected/3616efda-cc2c-4127-885e-49ea7d47fc12-kube-api-access-t7zr6\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284132    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284145    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284156    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284167    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3f5772-16c2-4650-8bd8-fdb0ec149974-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284179    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.284441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.284191    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.285765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.285743    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-envoy" pod "3616efda-cc2c-4127-885e-49ea7d47fc12" (UID: "3616efda-cc2c-4127-885e-49ea7d47fc12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:01.384968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.384943    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3616efda-cc2c-4127-885e-49ea7d47fc12-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:12:01.420791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.420759    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"]
Oct 30 13:12:01.424930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.424903    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh"]
Oct 30 13:12:01.876360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.876330    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-hxkhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:12:01.876855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:01.876387    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-hxkhh" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:02.076320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.076263    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"
Oct 30 13:12:02.077452 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.077424    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"
Oct 30 13:12:02.078140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.078119    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" path="/var/lib/kubelet/pods/9a3f5772-16c2-4650-8bd8-fdb0ec149974/volumes"
Oct 30 13:12:02.078731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.078713    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f" event={"ID":"3616efda-cc2c-4127-885e-49ea7d47fc12","Type":"ContainerDied","Data":"b49b3029f00e931baa77572acfa8ae9039116354dcd133e9ba2c5bc024ba7494"}
Oct 30 13:12:02.078816 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.078789    4429 scope.go:122] "RemoveContainer" containerID="40fffc686568d0dd8264e6c80807decd64715be0fe332256d692f81ad731d1db"
Oct 30 13:12:02.140821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.140763    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"]
Oct 30 13:12:02.156320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.156276    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-sjm4f"]
Oct 30 13:12:02.185371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.185345    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"]
Oct 30 13:12:02.190656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.190628    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-fks52"]
Oct 30 13:12:02.798167 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.798134    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"]
Oct 30 13:12:02.798853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.798833    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy"
Oct 30 13:12:02.798904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.798857    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy"
Oct 30 13:12:02.798904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.798871    4429 cpu_manager.go:414] "containerMap: removing container" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy"
Oct 30 13:12:02.798904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.798876    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy"
Oct 30 13:12:02.798904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.798890    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy"
Oct 30 13:12:02.798904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.798895    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy"
Oct 30 13:12:02.799120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.798998    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" containerName="istio-proxy"
Oct 30 13:12:02.799120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.799007    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3f5772-16c2-4650-8bd8-fdb0ec149974" containerName="istio-proxy"
Oct 30 13:12:02.799120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.799020    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" containerName="istio-proxy"
Oct 30 13:12:02.803724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.803706    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:02.831616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.831595    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-517a25\""
Oct 30 13:12:02.832765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.832747    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"]
Oct 30 13:12:02.916882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.916853    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabcd6da-a086-4864-b2e7-0d41500b6c83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq\" (UID: \"cabcd6da-a086-4864-b2e7-0d41500b6c83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:02.917307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.916924    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vkf7\" (UniqueName: \"kubernetes.io/projected/cabcd6da-a086-4864-b2e7-0d41500b6c83-kube-api-access-2vkf7\") pod \"payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq\" (UID: \"cabcd6da-a086-4864-b2e7-0d41500b6c83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:02.969968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.969937    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"]
Oct 30 13:12:02.984637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:02.984611    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:03.005498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.001790    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"]
Oct 30 13:12:03.011625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.011579    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-517a25-dockercfg-zhbcl\""
Oct 30 13:12:03.021572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.019491    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vk6b\" (UniqueName: \"kubernetes.io/projected/33e2e18d-e039-43aa-8822-52590f759cfd-kube-api-access-5vk6b\") pod \"payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp\" (UID: \"33e2e18d-e039-43aa-8822-52590f759cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:03.021572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.019559    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vkf7\" (UniqueName: \"kubernetes.io/projected/cabcd6da-a086-4864-b2e7-0d41500b6c83-kube-api-access-2vkf7\") pod \"payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq\" (UID: \"cabcd6da-a086-4864-b2e7-0d41500b6c83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:03.021572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.019680    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e2e18d-e039-43aa-8822-52590f759cfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp\" (UID: \"33e2e18d-e039-43aa-8822-52590f759cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:03.021572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.019871    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabcd6da-a086-4864-b2e7-0d41500b6c83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq\" (UID: \"cabcd6da-a086-4864-b2e7-0d41500b6c83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:03.021572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.020720    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabcd6da-a086-4864-b2e7-0d41500b6c83-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq\" (UID: \"cabcd6da-a086-4864-b2e7-0d41500b6c83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:03.068383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.066400    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vkf7\" (UniqueName: \"kubernetes.io/projected/cabcd6da-a086-4864-b2e7-0d41500b6c83-kube-api-access-2vkf7\") pod \"payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq\" (UID: \"cabcd6da-a086-4864-b2e7-0d41500b6c83\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:03.117304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.117259    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:03.120811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.120787    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e2e18d-e039-43aa-8822-52590f759cfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp\" (UID: \"33e2e18d-e039-43aa-8822-52590f759cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:03.121047 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.121029    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vk6b\" (UniqueName: \"kubernetes.io/projected/33e2e18d-e039-43aa-8822-52590f759cfd-kube-api-access-5vk6b\") pod \"payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp\" (UID: \"33e2e18d-e039-43aa-8822-52590f759cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:03.121711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.121678    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e2e18d-e039-43aa-8822-52590f759cfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp\" (UID: \"33e2e18d-e039-43aa-8822-52590f759cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:03.149183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.149158    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vk6b\" (UniqueName: \"kubernetes.io/projected/33e2e18d-e039-43aa-8822-52590f759cfd-kube-api-access-5vk6b\") pod \"payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp\" (UID: \"33e2e18d-e039-43aa-8822-52590f759cfd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:03.336754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.336653    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:03.438813 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:03.438781    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabcd6da_a086_4864_b2e7_0d41500b6c83.slice/crio-99406c0c6e8798790b73b4f83693380d731259fadb163433cece15bf652096ea WatchSource:0}: Error finding container 99406c0c6e8798790b73b4f83693380d731259fadb163433cece15bf652096ea: Status 404 returned error can't find the container with id 99406c0c6e8798790b73b4f83693380d731259fadb163433cece15bf652096ea
Oct 30 13:12:03.441195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.441172    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"]
Oct 30 13:12:03.449122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.449104    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:12:03.642717 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:03.642691    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e2e18d_e039_43aa_8822_52590f759cfd.slice/crio-eb8b8028d71725673e942759f2c890272702efbc572da15958faedb5ca440e66 WatchSource:0}: Error finding container eb8b8028d71725673e942759f2c890272702efbc572da15958faedb5ca440e66: Status 404 returned error can't find the container with id eb8b8028d71725673e942759f2c890272702efbc572da15958faedb5ca440e66
Oct 30 13:12:03.643672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:03.643595    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"]
Oct 30 13:12:04.078701 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.078647    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3616efda-cc2c-4127-885e-49ea7d47fc12" path="/var/lib/kubelet/pods/3616efda-cc2c-4127-885e-49ea7d47fc12/volumes"
Oct 30 13:12:04.079254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.079234    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb" path="/var/lib/kubelet/pods/b5229a88-b9ba-40ea-b08c-91dc4f1bc1fb/volumes"
Oct 30 13:12:04.093834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.093807    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp" event={"ID":"33e2e18d-e039-43aa-8822-52590f759cfd","Type":"ContainerStarted","Data":"b328e2572e1bdcf54db55100113f21b92f0db76b63254a895aa7ed16e102f79e"}
Oct 30 13:12:04.093983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.093857    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp" event={"ID":"33e2e18d-e039-43aa-8822-52590f759cfd","Type":"ContainerStarted","Data":"eb8b8028d71725673e942759f2c890272702efbc572da15958faedb5ca440e66"}
Oct 30 13:12:04.093983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.093946    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:04.095515 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.095491    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq" event={"ID":"cabcd6da-a086-4864-b2e7-0d41500b6c83","Type":"ContainerStarted","Data":"a1060cb4fc582028ced69e3694438f8749e3b23a50a253b88e517c384ddce6ea"}
Oct 30 13:12:04.095600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.095530    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq" event={"ID":"cabcd6da-a086-4864-b2e7-0d41500b6c83","Type":"ContainerStarted","Data":"99406c0c6e8798790b73b4f83693380d731259fadb163433cece15bf652096ea"}
Oct 30 13:12:04.095600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.095576    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:04.140471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.140434    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp" podStartSLOduration=2.140424062 podStartE2EDuration="2.140424062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:04.13715313 +0000 UTC m=+2382.192646955" watchObservedRunningTime="2026-10-30 13:12:04.140424062 +0000 UTC m=+2382.195917886"
Oct 30 13:12:04.183243 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.183205    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq" podStartSLOduration=2.183193439 podStartE2EDuration="2.183193439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:04.180515441 +0000 UTC m=+2382.236009267" watchObservedRunningTime="2026-10-30 13:12:04.183193439 +0000 UTC m=+2382.238687262"
Oct 30 13:12:04.380785 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.380696    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"]
Oct 30 13:12:04.386158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.386133    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.411331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.411022    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"]
Oct 30 13:12:04.439363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.439339    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.439524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.439503    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2tt\" (UniqueName: \"kubernetes.io/projected/edfc88d7-a686-4947-af56-ff979b8d603d-kube-api-access-wj2tt\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.439682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.439667    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.439716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.439697    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfc88d7-a686-4947-af56-ff979b8d603d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.458530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.458501    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e9cb4a-serving-cert\""
Oct 30 13:12:04.541552 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.541526    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.541676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.541561    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfc88d7-a686-4947-af56-ff979b8d603d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.541676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.541639    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.541676 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:12:04.541664    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-serving-cert: secret "maas-api-e2e-shared-a-w2-e9cb4a-serving-cert" not found
Oct 30 13:12:04.541831 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:12:04.541723    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls podName:edfc88d7-a686-4947-af56-ff979b8d603d nodeName:}" failed. No retries permitted until 2026-10-30 13:12:05.041707949 +0000 UTC m=+2383.097201764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls") pod "maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc" (UID: "edfc88d7-a686-4947-af56-ff979b8d603d") : secret "maas-api-e2e-shared-a-w2-e9cb4a-serving-cert" not found
Oct 30 13:12:04.541831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.541760    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2tt\" (UniqueName: \"kubernetes.io/projected/edfc88d7-a686-4947-af56-ff979b8d603d-kube-api-access-wj2tt\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.542397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.542379    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfc88d7-a686-4947-af56-ff979b8d603d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.544363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.544344    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:04.571619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:04.571584    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2tt\" (UniqueName: \"kubernetes.io/projected/edfc88d7-a686-4947-af56-ff979b8d603d-kube-api-access-wj2tt\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:05.047787 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:05.047755    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:05.050335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:05.050308    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:05.303213 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:05.303142    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:05.604203 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:05.604177    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfc88d7_a686_4947_af56_ff979b8d603d.slice/crio-ba523398c947454333ee46ddc11d3479e7fdabff4381b0132b5a40c5277be94e WatchSource:0}: Error finding container ba523398c947454333ee46ddc11d3479e7fdabff4381b0132b5a40c5277be94e: Status 404 returned error can't find the container with id ba523398c947454333ee46ddc11d3479e7fdabff4381b0132b5a40c5277be94e
Oct 30 13:12:05.606144 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:05.606121    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"]
Oct 30 13:12:05.606236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:05.606218    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:12:06.109116 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:06.109073    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc" event={"ID":"edfc88d7-a686-4947-af56-ff979b8d603d","Type":"ContainerStarted","Data":"d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60"}
Oct 30 13:12:06.109274 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:06.109139    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc" event={"ID":"edfc88d7-a686-4947-af56-ff979b8d603d","Type":"ContainerStarted","Data":"ba523398c947454333ee46ddc11d3479e7fdabff4381b0132b5a40c5277be94e"}
Oct 30 13:12:06.109365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:06.109311    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:06.163078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:06.163023    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc" podStartSLOduration=1.803069987 podStartE2EDuration="2.163002469s" totalImagesPullingTime="359.932482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:06.158848706 +0000 UTC m=+2384.214342531" watchObservedRunningTime="2026-10-30 13:12:06.163002469 +0000 UTC m=+2384.218496296"
Oct 30 13:12:12.121654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.121628    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:12:12.291821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.291786    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"]
Oct 30 13:12:12.298656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.298622    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.333654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.333629    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ca097a-openshift-default-dockercfg-btkhx\""
Oct 30 13:12:12.335335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.335311    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"]
Oct 30 13:12:12.411086 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:12:12.410839    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:12:12.480819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.480783    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-certs\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.480994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.480866    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-socket\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.480994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.480968    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istiod-ca-cert\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.481068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.481026    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-data\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.481132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.481113    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-envoy\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.481189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.481176    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-token\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.481244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.481228    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-credential-socket\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.481341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.481329    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-podinfo\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.481397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.481385    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7khh\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-kube-api-access-t7khh\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.582766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.582729    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-socket\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.582984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.582798    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istiod-ca-cert\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.582984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.582861    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-data\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.582984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.582900    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-envoy\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.582984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.582948    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-token\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583008    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-credential-socket\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583047    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-podinfo\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583132    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7khh\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-kube-api-access-t7khh\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583178    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-socket\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583192    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-certs\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583240    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-data\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583458    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-certs\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583607    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-credential-socket\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.583687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.583656    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istiod-ca-cert\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.586725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.586697    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-podinfo\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.591264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.591239    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-envoy\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.609177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.609153    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7khh\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-kube-api-access-t7khh\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.609428 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.609408    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-token\") pod \"e2e-praxis-b-ca097a-openshift-default-7478b86885-578km\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") " pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.612484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.612461    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:12.931354 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:12.931323    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bea9c3_b4f0_44be_a392_fa04d29a6dc7.slice/crio-73058e543e60f2e1312e619253b69b0a3f2e244d4dcbb489360343fa0b1f5805 WatchSource:0}: Error finding container 73058e543e60f2e1312e619253b69b0a3f2e244d4dcbb489360343fa0b1f5805: Status 404 returned error can't find the container with id 73058e543e60f2e1312e619253b69b0a3f2e244d4dcbb489360343fa0b1f5805
Oct 30 13:12:12.932531 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.932504    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"]
Oct 30 13:12:12.934118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.934087    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:12.934211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.934147    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:12.934211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:12.934175    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:13.165709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:13.165667    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" event={"ID":"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7","Type":"ContainerStarted","Data":"461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c"}
Oct 30 13:12:13.166230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:13.165739    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" event={"ID":"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7","Type":"ContainerStarted","Data":"73058e543e60f2e1312e619253b69b0a3f2e244d4dcbb489360343fa0b1f5805"}
Oct 30 13:12:13.223255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:13.223203    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" podStartSLOduration=1.223183179 podStartE2EDuration="1.223183179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:13.221062541 +0000 UTC m=+2391.276556378" watchObservedRunningTime="2026-10-30 13:12:13.223183179 +0000 UTC m=+2391.278677006"
Oct 30 13:12:13.613260 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:13.613195    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:14.619082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:14.619053    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:15.104967 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:15.104939    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:12:15.105113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:15.105010    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:12:15.178542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:15.178514    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:15.179773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:15.179751    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:12:16.668504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.668468    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"]
Oct 30 13:12:16.672902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.672880    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.714344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.700472    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"]
Oct 30 13:12:16.714344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.713775    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-517a25-serving-cert\""
Oct 30 13:12:16.746478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.746451    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be5a43a-2320-4e63-821a-e15e67da2f81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.746595 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.746493    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.746651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.746640    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25p2r\" (UniqueName: \"kubernetes.io/projected/6be5a43a-2320-4e63-821a-e15e67da2f81-kube-api-access-25p2r\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.746839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.746818    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.847721 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.847690    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.847893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.847743    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be5a43a-2320-4e63-821a-e15e67da2f81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.847893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.847770    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.847893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.847868    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25p2r\" (UniqueName: \"kubernetes.io/projected/6be5a43a-2320-4e63-821a-e15e67da2f81-kube-api-access-25p2r\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.848589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.848548    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be5a43a-2320-4e63-821a-e15e67da2f81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.850174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.850153    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.850367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.850348    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.873519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.873496    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25p2r\" (UniqueName: \"kubernetes.io/projected/6be5a43a-2320-4e63-821a-e15e67da2f81-kube-api-access-25p2r\") pod \"maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:16.989544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:16.989503    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:17.371988 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:17.371959    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be5a43a_2320_4e63_821a_e15e67da2f81.slice/crio-d0fbc3ef19abbdab09d8d38afc6a91b541049e7fa0bd3deaa7e4cabaee841700 WatchSource:0}: Error finding container d0fbc3ef19abbdab09d8d38afc6a91b541049e7fa0bd3deaa7e4cabaee841700: Status 404 returned error can't find the container with id d0fbc3ef19abbdab09d8d38afc6a91b541049e7fa0bd3deaa7e4cabaee841700
Oct 30 13:12:17.372869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:17.372848    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"]
Oct 30 13:12:18.230851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.229141    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8" event={"ID":"6be5a43a-2320-4e63-821a-e15e67da2f81","Type":"ContainerStarted","Data":"843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f"}
Oct 30 13:12:18.230851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.229207    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8" event={"ID":"6be5a43a-2320-4e63-821a-e15e67da2f81","Type":"ContainerStarted","Data":"d0fbc3ef19abbdab09d8d38afc6a91b541049e7fa0bd3deaa7e4cabaee841700"}
Oct 30 13:12:18.230851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.229649    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:18.280932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.280885    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8" podStartSLOduration=1.877584516 podStartE2EDuration="2.280868719s" totalImagesPullingTime="403.284203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:18.278595838 +0000 UTC m=+2396.334089659" watchObservedRunningTime="2026-10-30 13:12:18.280868719 +0000 UTC m=+2396.336362544"
Oct 30 13:12:18.776153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.776123    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"]
Oct 30 13:12:18.780535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.780515    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:18.790010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.789989    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l66l\" (UniqueName: \"kubernetes.io/projected/339c4cee-4f61-48ec-a96c-a5c2316edd02-kube-api-access-7l66l\") pod \"payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h\" (UID: \"339c4cee-4f61-48ec-a96c-a5c2316edd02\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:18.790110 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.790054    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c4cee-4f61-48ec-a96c-a5c2316edd02-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h\" (UID: \"339c4cee-4f61-48ec-a96c-a5c2316edd02\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:18.814536 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.814507    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"]
Oct 30 13:12:18.838738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.838712    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ca097a-dockercfg-cn9mm\""
Oct 30 13:12:18.847368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.847346    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ca097a\""
Oct 30 13:12:18.891079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.891045    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l66l\" (UniqueName: \"kubernetes.io/projected/339c4cee-4f61-48ec-a96c-a5c2316edd02-kube-api-access-7l66l\") pod \"payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h\" (UID: \"339c4cee-4f61-48ec-a96c-a5c2316edd02\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:18.891203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.891127    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c4cee-4f61-48ec-a96c-a5c2316edd02-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h\" (UID: \"339c4cee-4f61-48ec-a96c-a5c2316edd02\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:18.891844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.891814    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c4cee-4f61-48ec-a96c-a5c2316edd02-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h\" (UID: \"339c4cee-4f61-48ec-a96c-a5c2316edd02\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:18.945305 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:18.944572    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l66l\" (UniqueName: \"kubernetes.io/projected/339c4cee-4f61-48ec-a96c-a5c2316edd02-kube-api-access-7l66l\") pod \"payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h\" (UID: \"339c4cee-4f61-48ec-a96c-a5c2316edd02\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:19.064930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.064850    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"]
Oct 30 13:12:19.070008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.069983    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094297 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094250    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-podinfo\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094333    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-token\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094447    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-socket\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094511    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-credential-socket\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094543    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vtv\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-kube-api-access-k9vtv\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094619    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-data\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094711    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-envoy\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094750    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istiod-ca-cert\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.094904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.094879    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-certs\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.097417 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.097393    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:19.097867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.097848    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"]
Oct 30 13:12:19.113166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.113144    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-bd2aaf-openshift-default-dockercfg-zqp5h\""
Oct 30 13:12:19.337580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.309693    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-podinfo\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.341231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.341196    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-token\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.341431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.341308    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-socket\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.341431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.341346    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-credential-socket\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.341431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.341376    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vtv\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-kube-api-access-k9vtv\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.341605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.341437    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-data\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.341605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.341537    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-envoy\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.341718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.341606    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istiod-ca-cert\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.341782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.341758    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-certs\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.356615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.343263    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-socket\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.356615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.343631    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-credential-socket\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.356615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.344194    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-certs\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.356615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.344876    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istiod-ca-cert\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.356615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.345464    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-data\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.356615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.350093    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-podinfo\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.356615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.352148    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-envoy\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.438309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.432109    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vtv\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-kube-api-access-k9vtv\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.438309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.433441    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-token\") pod \"e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") " pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.438309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.437395    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:19.460889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.460675    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"]
Oct 30 13:12:19.773547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.773519    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"]
Oct 30 13:12:19.776143 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:19.776112    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153040de_ee6d_4eb0_9333_5e7d0c36dc60.slice/crio-899a2c646f8e7c1edc83f21d0181a16cd146a193ec98604b5c534b42e5ebd28d WatchSource:0}: Error finding container 899a2c646f8e7c1edc83f21d0181a16cd146a193ec98604b5c534b42e5ebd28d: Status 404 returned error can't find the container with id 899a2c646f8e7c1edc83f21d0181a16cd146a193ec98604b5c534b42e5ebd28d
Oct 30 13:12:19.779252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.779213    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:19.779373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.779349    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:19.779418 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:19.779399    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:20.405065 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.405026    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" event={"ID":"153040de-ee6d-4eb0-9333-5e7d0c36dc60","Type":"ContainerStarted","Data":"7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0"}
Oct 30 13:12:20.411535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.411502    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" event={"ID":"153040de-ee6d-4eb0-9333-5e7d0c36dc60","Type":"ContainerStarted","Data":"899a2c646f8e7c1edc83f21d0181a16cd146a193ec98604b5c534b42e5ebd28d"}
Oct 30 13:12:20.419021 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.418097    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h" event={"ID":"339c4cee-4f61-48ec-a96c-a5c2316edd02","Type":"ContainerStarted","Data":"101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b"}
Oct 30 13:12:20.419021 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.418152    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h" event={"ID":"339c4cee-4f61-48ec-a96c-a5c2316edd02","Type":"ContainerStarted","Data":"b4f7c582fb5bca7f1cd5a68c56a79b1392ea24ec119ce2bfc2c30504637160d2"}
Oct 30 13:12:20.419021 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.418957    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:20.466620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.453915    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:20.466620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.454041    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:12:20.466620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.454101    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 13:12:20.546004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:20.545938    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podStartSLOduration=1.545914942 podStartE2EDuration="1.545914942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:20.470589261 +0000 UTC m=+2398.526083086" watchObservedRunningTime="2026-10-30 13:12:20.545914942 +0000 UTC m=+2398.601408767"
Oct 30 13:12:21.443850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:21.443821    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:21.508919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:21.508869    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h" podStartSLOduration=3.508855015 podStartE2EDuration="3.508855015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:20.568402535 +0000 UTC m=+2398.623896398" watchObservedRunningTime="2026-10-30 13:12:21.508855015 +0000 UTC m=+2399.564348839"
Oct 30 13:12:22.432819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:22.432777    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:22.433649 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:22.433630    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:12:23.215770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:23.215734    4429 scope.go:122] "RemoveContainer" containerID="82e8d7aa3835a93308f4b670e847daca6053137d5a7c342ebfb4ed1b1d0d4c91"
Oct 30 13:12:24.374887 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:24.374856    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:12:29.505341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:29.505313    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" event={"ID":"922d2e60-03b5-4ae7-b439-3b57d47d2902","Type":"ContainerDied","Data":"eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f"}
Oct 30 13:12:29.505716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:29.505304    4429 generic.go:356] "Generic (PLEG): container finished" podID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerID="eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f" exitCode=0
Oct 30 13:12:29.505775 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:29.505759    4429 scope.go:122] "RemoveContainer" containerID="eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f"
Oct 30 13:12:29.507277 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:29.507247    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:29.507392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:29.507362    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:29.507432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:29.507393    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:30.514784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:30.514752    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" event={"ID":"922d2e60-03b5-4ae7-b439-3b57d47d2902","Type":"ContainerStarted","Data":"e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6"}
Oct 30 13:12:30.878869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:30.878712    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:12:30.886708 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:30.886386    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:12:30.886708 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:30.886440    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 13:12:31.870467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:31.870435    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:12:31.871026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:31.870995    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 13:12:32.439335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:32.437352    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:12:32.871845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:32.871750    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:12:32.872455 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:32.872424    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 13:12:33.390431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:33.390389    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:12:33.874739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:33.874702    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:12:33.875374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:33.875342    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 13:12:34.879310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:34.875739    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:12:34.879310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:34.875798    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 13:12:36.085647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:36.085618    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:12:36.086819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:36.086800    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:12:37.847323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:37.847145    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"]
Oct 30 13:12:37.874699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:37.874668    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:37.958570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:37.950328    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-98dabb-openshift-default-dockercfg-td6bk\""
Oct 30 13:12:38.000278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.000247    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"]
Oct 30 13:12:38.020046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020008    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-certs\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.020231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020077    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-credential-socket\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.020231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020208    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-podinfo\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.020367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020246    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-envoy\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.020422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020372    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqsk\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-kube-api-access-9kqsk\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.020422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020406    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6206414c-fa76-42c9-a13e-7d4a45fea76b-istiod-ca-cert\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.020531 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020473    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-socket\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.020585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020528    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-token\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.020585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.020560    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-data\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122020 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.121915    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqsk\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-kube-api-access-9kqsk\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122020 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.121971    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6206414c-fa76-42c9-a13e-7d4a45fea76b-istiod-ca-cert\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.122080    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-socket\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.122175    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-token\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.122380    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-data\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.122437    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-certs\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.122491    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-credential-socket\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.122562    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-podinfo\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.122751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.122596    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-envoy\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.123606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.123367    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-data\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.124410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.124366    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6206414c-fa76-42c9-a13e-7d4a45fea76b-istiod-ca-cert\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.125178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.124732    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-socket\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.125178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.124834    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-certs\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.125178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.125087    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-credential-socket\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.128875 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.128856    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-envoy\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.129014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.128993    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-podinfo\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.187131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.184196    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqsk\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-kube-api-access-9kqsk\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.190106 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.190077    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-token\") pod \"e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") " pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.313783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.313745    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:38.671176 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:38.671142    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6206414c_fa76_42c9_a13e_7d4a45fea76b.slice/crio-0718df1ee51d52faaba858e200c1dc7092ad9af319a4b6887585741250128ebf WatchSource:0}: Error finding container 0718df1ee51d52faaba858e200c1dc7092ad9af319a4b6887585741250128ebf: Status 404 returned error can't find the container with id 0718df1ee51d52faaba858e200c1dc7092ad9af319a4b6887585741250128ebf
Oct 30 13:12:38.671642 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.671614    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"]
Oct 30 13:12:38.674199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.674166    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:38.674313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.674233    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:38.674313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:38.674264    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:39.133595 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.133559    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"]
Oct 30 13:12:39.138160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.138138    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:39.156836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.156784    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-92855c\""
Oct 30 13:12:39.161108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.160918    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"]
Oct 30 13:12:39.189391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.189360    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"]
Oct 30 13:12:39.194637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.194617    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:39.221142 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.221117    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"]
Oct 30 13:12:39.222181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.222163    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-92855c-dockercfg-c7hwq\""
Oct 30 13:12:39.239061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.239039    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b724c857-22c4-41c1-a1f4-85548435e237-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52\" (UID: \"b724c857-22c4-41c1-a1f4-85548435e237\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:39.239185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.239171    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k4dj\" (UniqueName: \"kubernetes.io/projected/b724c857-22c4-41c1-a1f4-85548435e237-kube-api-access-7k4dj\") pod \"payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52\" (UID: \"b724c857-22c4-41c1-a1f4-85548435e237\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:39.239260 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.239193    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zztm\" (UniqueName: \"kubernetes.io/projected/bc7d3080-95f3-4fca-9680-24a2b982f3d0-kube-api-access-8zztm\") pod \"payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx\" (UID: \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:39.239260 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.239239    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7d3080-95f3-4fca-9680-24a2b982f3d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx\" (UID: \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:39.343348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.343276    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zztm\" (UniqueName: \"kubernetes.io/projected/bc7d3080-95f3-4fca-9680-24a2b982f3d0-kube-api-access-8zztm\") pod \"payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx\" (UID: \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:39.344088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.344062    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7d3080-95f3-4fca-9680-24a2b982f3d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx\" (UID: \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:39.345323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.345301    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b724c857-22c4-41c1-a1f4-85548435e237-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52\" (UID: \"b724c857-22c4-41c1-a1f4-85548435e237\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:39.348332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.348278    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k4dj\" (UniqueName: \"kubernetes.io/projected/b724c857-22c4-41c1-a1f4-85548435e237-kube-api-access-7k4dj\") pod \"payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52\" (UID: \"b724c857-22c4-41c1-a1f4-85548435e237\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:39.348580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.347675    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b724c857-22c4-41c1-a1f4-85548435e237-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52\" (UID: \"b724c857-22c4-41c1-a1f4-85548435e237\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:39.348715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.345114    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7d3080-95f3-4fca-9680-24a2b982f3d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx\" (UID: \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:39.392403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.382252    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zztm\" (UniqueName: \"kubernetes.io/projected/bc7d3080-95f3-4fca-9680-24a2b982f3d0-kube-api-access-8zztm\") pod \"payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx\" (UID: \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:39.409438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.409357    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k4dj\" (UniqueName: \"kubernetes.io/projected/b724c857-22c4-41c1-a1f4-85548435e237-kube-api-access-7k4dj\") pod \"payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52\" (UID: \"b724c857-22c4-41c1-a1f4-85548435e237\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:39.456199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.456165    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:39.515517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.515449    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:39.618605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.618569    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" event={"ID":"6206414c-fa76-42c9-a13e-7d4a45fea76b","Type":"ContainerStarted","Data":"69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc"}
Oct 30 13:12:39.618727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.618676    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" event={"ID":"6206414c-fa76-42c9-a13e-7d4a45fea76b","Type":"ContainerStarted","Data":"0718df1ee51d52faaba858e200c1dc7092ad9af319a4b6887585741250128ebf"}
Oct 30 13:12:39.679883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.678303    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" podStartSLOduration=2.678267059 podStartE2EDuration="2.678267059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:39.672603073 +0000 UTC m=+2417.728096926" watchObservedRunningTime="2026-10-30 13:12:39.678267059 +0000 UTC m=+2417.733760881"
Oct 30 13:12:39.766731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.766701    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"]
Oct 30 13:12:39.767569 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:39.767538    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb724c857_22c4_41c1_a1f4_85548435e237.slice/crio-e4aea6bdf2eb18c397573cd465c7f280dded863c6c535716db8586a43bc3017b WatchSource:0}: Error finding container e4aea6bdf2eb18c397573cd465c7f280dded863c6c535716db8586a43bc3017b: Status 404 returned error can't find the container with id e4aea6bdf2eb18c397573cd465c7f280dded863c6c535716db8586a43bc3017b
Oct 30 13:12:39.914617 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:39.914580    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7d3080_95f3_4fca_9680_24a2b982f3d0.slice/crio-91b8ac898258bfd11130fb5cf30ae40ef5426fa8aece36cf0c318008effee404 WatchSource:0}: Error finding container 91b8ac898258bfd11130fb5cf30ae40ef5426fa8aece36cf0c318008effee404: Status 404 returned error can't find the container with id 91b8ac898258bfd11130fb5cf30ae40ef5426fa8aece36cf0c318008effee404
Oct 30 13:12:39.915328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:39.915296    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"]
Oct 30 13:12:40.314341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.314299    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:40.320612 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.320584    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:40.493506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.493466    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:12:40.493722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.493541    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:40.628007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.627920    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52" event={"ID":"b724c857-22c4-41c1-a1f4-85548435e237","Type":"ContainerStarted","Data":"abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd"}
Oct 30 13:12:40.628007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.627983    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:40.628007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.628006    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52" event={"ID":"b724c857-22c4-41c1-a1f4-85548435e237","Type":"ContainerStarted","Data":"e4aea6bdf2eb18c397573cd465c7f280dded863c6c535716db8586a43bc3017b"}
Oct 30 13:12:40.629903 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.629870    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx" event={"ID":"bc7d3080-95f3-4fca-9680-24a2b982f3d0","Type":"ContainerStarted","Data":"3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6"}
Oct 30 13:12:40.630049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.629929    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx" event={"ID":"bc7d3080-95f3-4fca-9680-24a2b982f3d0","Type":"ContainerStarted","Data":"91b8ac898258bfd11130fb5cf30ae40ef5426fa8aece36cf0c318008effee404"}
Oct 30 13:12:40.630165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.630149    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:40.630232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.630195    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:40.631138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.631115    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:12:40.675164 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.675112    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52" podStartSLOduration=1.6750952460000001 podStartE2EDuration="1.675095246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:40.671720167 +0000 UTC m=+2418.727214009" watchObservedRunningTime="2026-10-30 13:12:40.675095246 +0000 UTC m=+2418.730589070"
Oct 30 13:12:40.718592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:40.718538    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx" podStartSLOduration=1.718523566 podStartE2EDuration="1.718523566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:40.715491191 +0000 UTC m=+2418.770985014" watchObservedRunningTime="2026-10-30 13:12:40.718523566 +0000 UTC m=+2418.774017390"
Oct 30 13:12:42.372270 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:12:42.372221    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:12:43.689018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:43.688984    4429 generic.go:356] "Generic (PLEG): container finished" podID="b327a616-e506-492f-88b7-66e4143df16a" containerID="63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6" exitCode=0
Oct 30 13:12:43.689561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:43.689056    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" event={"ID":"b327a616-e506-492f-88b7-66e4143df16a","Type":"ContainerDied","Data":"63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6"}
Oct 30 13:12:43.689561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:43.689543    4429 scope.go:122] "RemoveContainer" containerID="63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6"
Oct 30 13:12:43.691097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:43.691074    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:43.691158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:43.691141    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:43.691192 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:43.691174    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:44.077609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.077536    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"]
Oct 30 13:12:44.081828 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.081808    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:44.103185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.103163    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"]
Oct 30 13:12:44.113153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.113131    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-98dabb\""
Oct 30 13:12:44.113400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.113382    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-98dabb-dockercfg-964c5\""
Oct 30 13:12:44.120994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.120960    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjt5s\" (UniqueName: \"kubernetes.io/projected/591c870e-000d-4d4d-a1a6-1f52be1a23da-kube-api-access-kjt5s\") pod \"payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69\" (UID: \"591c870e-000d-4d4d-a1a6-1f52be1a23da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:44.121126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.121108    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c870e-000d-4d4d-a1a6-1f52be1a23da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69\" (UID: \"591c870e-000d-4d4d-a1a6-1f52be1a23da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:44.229376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.229331    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c870e-000d-4d4d-a1a6-1f52be1a23da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69\" (UID: \"591c870e-000d-4d4d-a1a6-1f52be1a23da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:44.229637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.229618    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjt5s\" (UniqueName: \"kubernetes.io/projected/591c870e-000d-4d4d-a1a6-1f52be1a23da-kube-api-access-kjt5s\") pod \"payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69\" (UID: \"591c870e-000d-4d4d-a1a6-1f52be1a23da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:44.230921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.230892    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c870e-000d-4d4d-a1a6-1f52be1a23da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69\" (UID: \"591c870e-000d-4d4d-a1a6-1f52be1a23da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:44.293962 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.293884    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjt5s\" (UniqueName: \"kubernetes.io/projected/591c870e-000d-4d4d-a1a6-1f52be1a23da-kube-api-access-kjt5s\") pod \"payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69\" (UID: \"591c870e-000d-4d4d-a1a6-1f52be1a23da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:44.396401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.396325    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:44.698107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.698070    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" event={"ID":"b327a616-e506-492f-88b7-66e4143df16a","Type":"ContainerStarted","Data":"5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764"}
Oct 30 13:12:44.716235 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:44.716194    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591c870e_000d_4d4d_a1a6_1f52be1a23da.slice/crio-4b7c5f261d8a0a380a11a1268c89af0eb8fda056c4da11b93b03a4592e2f3152 WatchSource:0}: Error finding container 4b7c5f261d8a0a380a11a1268c89af0eb8fda056c4da11b93b03a4592e2f3152: Status 404 returned error can't find the container with id 4b7c5f261d8a0a380a11a1268c89af0eb8fda056c4da11b93b03a4592e2f3152
Oct 30 13:12:44.717851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:44.717830    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"]
Oct 30 13:12:45.065367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:45.063677    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:12:45.065367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:45.063805    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:12:45.065367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:45.063847    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 13:12:45.706911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:45.706877    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69" event={"ID":"591c870e-000d-4d4d-a1a6-1f52be1a23da","Type":"ContainerStarted","Data":"37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769"}
Oct 30 13:12:45.707426 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:45.706961    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:45.707426 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:45.706978    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69" event={"ID":"591c870e-000d-4d4d-a1a6-1f52be1a23da","Type":"ContainerStarted","Data":"4b7c5f261d8a0a380a11a1268c89af0eb8fda056c4da11b93b03a4592e2f3152"}
Oct 30 13:12:45.775380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:45.775328    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69" podStartSLOduration=1.775314414 podStartE2EDuration="1.775314414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:45.770539993 +0000 UTC m=+2423.826033818" watchObservedRunningTime="2026-10-30 13:12:45.775314414 +0000 UTC m=+2423.830808237"
Oct 30 13:12:46.072390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:46.061624    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:12:46.072390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:46.061684    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 13:12:47.066310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:47.065731    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:12:47.066310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:47.065798    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 13:12:48.071310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:48.069127    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:12:48.071310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:48.069180    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 13:12:49.060655 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:49.060622    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 13:12:49.060949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:49.060915    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 13:12:50.126737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:50.126696    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:12:50.127345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:50.127155    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:12:50.128048 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:50.128029    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:12:51.681879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:51.681853    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:12:51.682492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:51.681922    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:12:52.815056 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:52.815022    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"]
Oct 30 13:12:52.819672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:52.819654    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:52.847006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:52.846983    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-241ffa\""
Oct 30 13:12:52.871308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:52.865503    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-241ffa-dockercfg-shrph\""
Oct 30 13:12:52.871308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:52.867161    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"]
Oct 30 13:12:52.981010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:52.980812    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g5zg\" (UniqueName: \"kubernetes.io/projected/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-kube-api-access-8g5zg\") pod \"payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6\" (UID: \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:52.981425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:52.981402    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6\" (UID: \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:53.086785 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.086696    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g5zg\" (UniqueName: \"kubernetes.io/projected/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-kube-api-access-8g5zg\") pod \"payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6\" (UID: \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:53.087408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.087380    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6\" (UID: \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:53.088517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.088373    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6\" (UID: \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:53.135073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.135046    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g5zg\" (UniqueName: \"kubernetes.io/projected/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-kube-api-access-8g5zg\") pod \"payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6\" (UID: \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:53.433953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.433921    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:53.777161 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:53.777127    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5b7987_fc95_4e40_9acf_6f4b52b68a8d.slice/crio-0f076d0567e90ccc4704474216d0236aeade0e3c2eeb00ffbe9b4c630c4d542e WatchSource:0}: Error finding container 0f076d0567e90ccc4704474216d0236aeade0e3c2eeb00ffbe9b4c630c4d542e: Status 404 returned error can't find the container with id 0f076d0567e90ccc4704474216d0236aeade0e3c2eeb00ffbe9b4c630c4d542e
Oct 30 13:12:53.777653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.777630    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"]
Oct 30 13:12:53.894528 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.894494    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6" event={"ID":"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d","Type":"ContainerStarted","Data":"a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c"}
Oct 30 13:12:53.894867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.894557    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6" event={"ID":"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d","Type":"ContainerStarted","Data":"0f076d0567e90ccc4704474216d0236aeade0e3c2eeb00ffbe9b4c630c4d542e"}
Oct 30 13:12:53.894867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.894612    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:12:53.940340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:53.940279    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6" podStartSLOduration=1.94026715 podStartE2EDuration="1.94026715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:53.938566968 +0000 UTC m=+2431.994060807" watchObservedRunningTime="2026-10-30 13:12:53.94026715 +0000 UTC m=+2431.995760973"
Oct 30 13:12:56.723711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.723676    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"]
Oct 30 13:12:56.728122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.728095    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.731334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.731313    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:12:56.759456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.759424    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"]
Oct 30 13:12:56.759749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.759734    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-92855c-serving-cert\""
Oct 30 13:12:56.853923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.853882    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc24016f-ddfd-4c79-a015-4684b56b817f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.854101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.854013    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5jpw\" (UniqueName: \"kubernetes.io/projected/bc24016f-ddfd-4c79-a015-4684b56b817f-kube-api-access-q5jpw\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.854171 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.854124    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.854317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.854275    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.961380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.961192    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5jpw\" (UniqueName: \"kubernetes.io/projected/bc24016f-ddfd-4c79-a015-4684b56b817f-kube-api-access-q5jpw\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.961380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.961325    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.961663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.961427    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.961663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.961503    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc24016f-ddfd-4c79-a015-4684b56b817f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.962549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.962517    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc24016f-ddfd-4c79-a015-4684b56b817f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.972124 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:12:56.971883    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-serving-cert: secret "maas-api-e2e-shared-b-w2-92855c-serving-cert" not found
Oct 30 13:12:56.972124 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:12:56.971973    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls podName:bc24016f-ddfd-4c79-a015-4684b56b817f nodeName:}" failed. No retries permitted until 2026-10-30 13:12:57.471952521 +0000 UTC m=+2435.527446323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls") pod "maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts" (UID: "bc24016f-ddfd-4c79-a015-4684b56b817f") : secret "maas-api-e2e-shared-b-w2-92855c-serving-cert" not found
Oct 30 13:12:56.991376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.981865    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:56.998478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:56.998447    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5jpw\" (UniqueName: \"kubernetes.io/projected/bc24016f-ddfd-4c79-a015-4684b56b817f-kube-api-access-q5jpw\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:57.569650 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:57.569609    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:57.573870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:57.573838    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:57.649022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:57.648996    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:57.971220 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:12:57.971192    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc24016f_ddfd_4c79_a015_4684b56b817f.slice/crio-70f474e1abd9de04c93efcd0f011f47c70cae7416c224ba47efd82d83644eeff WatchSource:0}: Error finding container 70f474e1abd9de04c93efcd0f011f47c70cae7416c224ba47efd82d83644eeff: Status 404 returned error can't find the container with id 70f474e1abd9de04c93efcd0f011f47c70cae7416c224ba47efd82d83644eeff
Oct 30 13:12:57.971999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:57.971974    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"]
Oct 30 13:12:58.935634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.935602    4429 generic.go:356] "Generic (PLEG): container finished" podID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerID="461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c" exitCode=0
Oct 30 13:12:58.935849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.935679    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" event={"ID":"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7","Type":"ContainerDied","Data":"461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c"}
Oct 30 13:12:58.936182 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.936153    4429 scope.go:122] "RemoveContainer" containerID="461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c"
Oct 30 13:12:58.937659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.937633    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts" event={"ID":"bc24016f-ddfd-4c79-a015-4684b56b817f","Type":"ContainerStarted","Data":"249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6"}
Oct 30 13:12:58.937811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.937686    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts" event={"ID":"bc24016f-ddfd-4c79-a015-4684b56b817f","Type":"ContainerStarted","Data":"70f474e1abd9de04c93efcd0f011f47c70cae7416c224ba47efd82d83644eeff"}
Oct 30 13:12:58.937874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.937849    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:12:58.938234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.938204    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:58.938332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.938314    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:58.938388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:58.938372    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:59.057632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:59.057569    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts" podStartSLOduration=2.601966451 podStartE2EDuration="3.057547839s" totalImagesPullingTime="455.581388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:59.05660564 +0000 UTC m=+2437.112099483" watchObservedRunningTime="2026-10-30 13:12:59.057547839 +0000 UTC m=+2437.113041667"
Oct 30 13:12:59.946254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:12:59.946216    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" event={"ID":"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7","Type":"ContainerStarted","Data":"1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f"}
Oct 30 13:13:00.179402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:00.179367    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:13:00.612930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:00.612898    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:13:00.613965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:00.613935    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:13:00.614104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:00.614003    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:13:01.613418 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:01.613386    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:13:01.613836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:01.613441    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:13:02.613463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:02.613429    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:13:02.613885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:02.613488    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:13:03.613755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:03.613722    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:13:03.614139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:03.613813    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:13:04.613556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:04.613519    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:13:04.613772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:04.613596    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 13:13:04.904976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:04.904899    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:13:04.952873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:04.952388    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:13:05.617998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:05.617963    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:13:05.618931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:05.618910    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:13:08.908177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.908138    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"]
Oct 30 13:13:08.913879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.913854    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:08.935941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.935912    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"]
Oct 30 13:13:08.944552 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.944521    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-2fdf91\"/\"kube-root-ca.crt\""
Oct 30 13:13:08.944702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.944623    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-2fdf91\"/\"default-dockercfg-6fsn8\""
Oct 30 13:13:08.944777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.944719    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-2fdf91\"/\"praxis-route-2fdf91-kserve-self-signed-certs\""
Oct 30 13:13:08.944836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.944820    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-2fdf91\"/\"openshift-service-ca.crt\""
Oct 30 13:13:08.996821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.996790    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-home\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:08.997028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.996854    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5l9\" (UniqueName: \"kubernetes.io/projected/0116a939-fad3-43ad-8e7a-4d4ec958c685-kube-api-access-kk5l9\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:08.997028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.996974    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-dshm\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:08.997028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.997017    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-tmp-dir\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:08.997184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.997160    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0116a939-fad3-43ad-8e7a-4d4ec958c685-tls-certs\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:08.997309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:08.997266    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-model-cache\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098269    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-dshm\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098335    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-tmp-dir\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098406    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0116a939-fad3-43ad-8e7a-4d4ec958c685-tls-certs\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098459    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-model-cache\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098651    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-home\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098737    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5l9\" (UniqueName: \"kubernetes.io/projected/0116a939-fad3-43ad-8e7a-4d4ec958c685-kube-api-access-kk5l9\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098827 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098776    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-tmp-dir\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098879    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-model-cache\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.098968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.098946    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-home\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.100823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.100802    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-dshm\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.100984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.100967    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0116a939-fad3-43ad-8e7a-4d4ec958c685-tls-certs\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.125946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.125924    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5l9\" (UniqueName: \"kubernetes.io/projected/0116a939-fad3-43ad-8e7a-4d4ec958c685-kube-api-access-kk5l9\") pod \"praxis-route-2fdf91-kserve-79d7b59477-q8dvv\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") " pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.231549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.231509    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:09.551179 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:13:09.551155    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0116a939_fad3_43ad_8e7a_4d4ec958c685.slice/crio-2d533b34a8a13f01ab4903e4907e890d90fb49d19b3b2328464a279c53ecf0e1 WatchSource:0}: Error finding container 2d533b34a8a13f01ab4903e4907e890d90fb49d19b3b2328464a279c53ecf0e1: Status 404 returned error can't find the container with id 2d533b34a8a13f01ab4903e4907e890d90fb49d19b3b2328464a279c53ecf0e1
Oct 30 13:13:09.552343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:09.552311    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"]
Oct 30 13:13:10.049800 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:10.049759    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv" event={"ID":"0116a939-fad3-43ad-8e7a-4d4ec958c685","Type":"ContainerStarted","Data":"f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929"}
Oct 30 13:13:10.050298 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:10.049812    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv" event={"ID":"0116a939-fad3-43ad-8e7a-4d4ec958c685","Type":"ContainerStarted","Data":"2d533b34a8a13f01ab4903e4907e890d90fb49d19b3b2328464a279c53ecf0e1"}
Oct 30 13:13:10.102477 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:10.102423    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv" podStartSLOduration=2.102408342 podStartE2EDuration="2.102408342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:10.100755805 +0000 UTC m=+2448.156249647" watchObservedRunningTime="2026-10-30 13:13:10.102408342 +0000 UTC m=+2448.157902203"
Oct 30 13:13:10.245955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:10.234427    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:12.374223 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:13:12.374182    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:13:19.232347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:19.232305    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:19.245480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:19.245451    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:20.143431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:20.143401    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:13:21.040047 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.040017    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"]
Oct 30 13:13:21.045558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.045534    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:21.064002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.063972    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"]
Oct 30 13:13:21.072544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.072514    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-71fbeb\""
Oct 30 13:13:21.072544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.072538    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-71fbeb-dockercfg-6tjdg\""
Oct 30 13:13:21.155318 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.153770    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjhl\" (UniqueName: \"kubernetes.io/projected/62d4287c-8aba-4241-9d46-66577c5dc2b6-kube-api-access-nqjhl\") pod \"payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw\" (UID: \"62d4287c-8aba-4241-9d46-66577c5dc2b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:21.155318 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.153857    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d4287c-8aba-4241-9d46-66577c5dc2b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw\" (UID: \"62d4287c-8aba-4241-9d46-66577c5dc2b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:21.288486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.285145    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjhl\" (UniqueName: \"kubernetes.io/projected/62d4287c-8aba-4241-9d46-66577c5dc2b6-kube-api-access-nqjhl\") pod \"payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw\" (UID: \"62d4287c-8aba-4241-9d46-66577c5dc2b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:21.292900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.288728    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d4287c-8aba-4241-9d46-66577c5dc2b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw\" (UID: \"62d4287c-8aba-4241-9d46-66577c5dc2b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:21.304255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.299965    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d4287c-8aba-4241-9d46-66577c5dc2b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw\" (UID: \"62d4287c-8aba-4241-9d46-66577c5dc2b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:21.319266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.319236    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjhl\" (UniqueName: \"kubernetes.io/projected/62d4287c-8aba-4241-9d46-66577c5dc2b6-kube-api-access-nqjhl\") pod \"payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw\" (UID: \"62d4287c-8aba-4241-9d46-66577c5dc2b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:21.363300 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.363258    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:21.701337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:21.700766    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"]
Oct 30 13:13:22.315633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:22.315589    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw" event={"ID":"62d4287c-8aba-4241-9d46-66577c5dc2b6","Type":"ContainerStarted","Data":"b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb"}
Oct 30 13:13:22.316129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:22.315657    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw" event={"ID":"62d4287c-8aba-4241-9d46-66577c5dc2b6","Type":"ContainerStarted","Data":"55c27076d3e69f1251eecc2613ea9967abd697fae5f334e9824b44b5f71c1b87"}
Oct 30 13:13:22.316129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:22.315706    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:22.366424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:22.366363    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw" podStartSLOduration=1.366347018 podStartE2EDuration="1.366347018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:22.363053469 +0000 UTC m=+2460.418547294" watchObservedRunningTime="2026-10-30 13:13:22.366347018 +0000 UTC m=+2460.421840844"
Oct 30 13:13:23.029183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:23.027618    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": read tcp 10.133.0.2:58954->10.133.0.149:15021: read: connection reset by peer" start-of-body=
Oct 30 13:13:23.029183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:23.027685    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": read tcp 10.133.0.2:58954->10.133.0.149:15021: read: connection reset by peer"
Oct 30 13:13:23.322934 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:23.322849    4429 generic.go:356] "Generic (PLEG): container finished" podID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerID="7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0" exitCode=0
Oct 30 13:13:23.322934 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:23.322917    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" event={"ID":"153040de-ee6d-4eb0-9333-5e7d0c36dc60","Type":"ContainerDied","Data":"7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0"}
Oct 30 13:13:23.323422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:23.323389    4429 scope.go:122] "RemoveContainer" containerID="7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0"
Oct 30 13:13:23.325081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:23.325054    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:23.325160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:23.325140    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:23.325200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:23.325176    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:24.331689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:24.331656    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" event={"ID":"153040de-ee6d-4eb0-9333-5e7d0c36dc60","Type":"ContainerStarted","Data":"491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9"}
Oct 30 13:13:24.438609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:24.438579    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:13:24.438897 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:24.438868    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:13:24.439016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:24.438925    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 13:13:25.438557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:25.438518    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:13:25.438949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:25.438578    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 13:13:26.438683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:26.438645    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:13:26.439089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:26.438702    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 13:13:27.438998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:27.438963    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:13:27.439693 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:27.439028    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 13:13:28.438661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:28.438624    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:13:28.438855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:28.438682    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 13:13:29.438910 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:29.438879    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:13:29.439338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:29.438927    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 13:13:30.442592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:30.442563    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:13:30.443069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:30.443049    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:13:30.443801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:30.443784    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:13:33.327033 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:33.326958    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:13:40.632000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:40.631969    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 13:13:40.632381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:40.632017    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 13:13:41.482067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:41.482037    4429 generic.go:356] "Generic (PLEG): container finished" podID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerID="69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc" exitCode=0
Oct 30 13:13:41.482238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:41.482115    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" event={"ID":"6206414c-fa76-42c9-a13e-7d4a45fea76b","Type":"ContainerDied","Data":"69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc"}
Oct 30 13:13:41.482604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:41.482588    4429 scope.go:122] "RemoveContainer" containerID="69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc"
Oct 30 13:13:41.484392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:41.484364    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:41.484470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:41.484437    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:41.484510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:41.484468    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:42.396994 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:13:42.396959    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:13:42.491652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:42.491617    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" event={"ID":"6206414c-fa76-42c9-a13e-7d4a45fea76b","Type":"ContainerStarted","Data":"565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4"}
Oct 30 13:13:43.314106 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:43.314069    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:13:43.314424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:43.314401    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 13:13:43.314487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:43.314449    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 13:13:44.339235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:44.334715    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 13:13:44.339235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:44.334778    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 13:13:45.315110 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:45.315077    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 13:13:45.315304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:45.315135    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 13:13:46.289365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.289324    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"]
Oct 30 13:13:46.294273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.294250    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.315253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.314980    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 13:13:46.315520 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.315360    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 13:13:46.323204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.323162    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.323433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.323233    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5l9w\" (UniqueName: \"kubernetes.io/projected/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-kube-api-access-k5l9w\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.323563 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.323541    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.323868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.323829    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.333579 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.333541    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-71fbeb-serving-cert\""
Oct 30 13:13:46.335034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.335005    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"]
Oct 30 13:13:46.425921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.425878    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.426108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.425946    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5l9w\" (UniqueName: \"kubernetes.io/projected/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-kube-api-access-k5l9w\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.426155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.426130    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.426223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.426203    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.426431 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:13:46.426414    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-serving-cert: secret "maas-api-e2e-shared-b-w1-71fbeb-serving-cert" not found
Oct 30 13:13:46.426505 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:13:46.426493    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls podName:60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9 nodeName:}" failed. No retries permitted until 2026-10-30 13:13:46.926472548 +0000 UTC m=+2484.981966352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls") pod "maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb" (UID: "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9") : secret "maas-api-e2e-shared-b-w1-71fbeb-serving-cert" not found
Oct 30 13:13:46.426948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.426929    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.428911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.428885    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.476098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.476060    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5l9w\" (UniqueName: \"kubernetes.io/projected/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-kube-api-access-k5l9w\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.934065 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.934031    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:46.936587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:46.936563    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:47.209507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:47.209397    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:47.317601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:47.317521    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 13:13:47.317601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:47.317582    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 13:13:47.554821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:47.554787    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"]
Oct 30 13:13:47.556445 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:13:47.556419    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e4eccb_a6cc_462f_8a8c_e6d5db3f00c9.slice/crio-1bc84a4e76676bf90d2ad716eebd76f31a1ee7f75cdfc23a44234b142534ce3d WatchSource:0}: Error finding container 1bc84a4e76676bf90d2ad716eebd76f31a1ee7f75cdfc23a44234b142534ce3d: Status 404 returned error can't find the container with id 1bc84a4e76676bf90d2ad716eebd76f31a1ee7f75cdfc23a44234b142534ce3d
Oct 30 13:13:48.319375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:48.319349    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:13:48.319772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:48.319670    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:13:48.320396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:48.320378    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:13:48.536454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:48.536415    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb" event={"ID":"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9","Type":"ContainerStarted","Data":"71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc"}
Oct 30 13:13:48.536640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:48.536473    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb" event={"ID":"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9","Type":"ContainerStarted","Data":"1bc84a4e76676bf90d2ad716eebd76f31a1ee7f75cdfc23a44234b142534ce3d"}
Oct 30 13:13:48.536706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:48.536688    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:13:48.585650 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:48.585542    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb" podStartSLOduration=2.331907636 podStartE2EDuration="2.585412787s" totalImagesPullingTime="253.505151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:48.582582451 +0000 UTC m=+2486.638076276" watchObservedRunningTime="2026-10-30 13:13:48.585412787 +0000 UTC m=+2486.640906615"
Oct 30 13:13:54.551255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:13:54.551223    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:14:04.610370 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.610338    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"]
Oct 30 13:14:04.616523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.616500    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.639565 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.639540    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"]
Oct 30 13:14:04.648793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.648772    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-517a25\"/\"default-dockercfg-spgj5\""
Oct 30 13:14:04.648930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.648869    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-517a25\"/\"auth-test-model-517a25-kserve-self-signed-certs\""
Oct 30 13:14:04.649010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.648960    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-517a25\"/\"openshift-service-ca.crt\""
Oct 30 13:14:04.649067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.649023    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-517a25\"/\"kube-root-ca.crt\""
Oct 30 13:14:04.703263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.703235    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0f79fe-2234-47ab-bee0-7301c97a6104-tls-certs\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.703462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.703443    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8q56\" (UniqueName: \"kubernetes.io/projected/fc0f79fe-2234-47ab-bee0-7301c97a6104-kube-api-access-p8q56\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.703529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.703477    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-dshm\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.703625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.703606    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-home\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.703790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.703772    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-model-cache\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.703837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.703811    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-tmp-dir\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.805332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805309    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-model-cache\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.805447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805344    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-tmp-dir\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.805506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805485    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0f79fe-2234-47ab-bee0-7301c97a6104-tls-certs\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.805624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805608    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8q56\" (UniqueName: \"kubernetes.io/projected/fc0f79fe-2234-47ab-bee0-7301c97a6104-kube-api-access-p8q56\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.805683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805649    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-dshm\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.805742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805726    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-tmp-dir\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.805796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805730    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-home\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.805796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805772    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-model-cache\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.806017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.805978    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-home\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.808036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.808018    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-dshm\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.808214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.808197    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0f79fe-2234-47ab-bee0-7301c97a6104-tls-certs\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.830919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.830900    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8q56\" (UniqueName: \"kubernetes.io/projected/fc0f79fe-2234-47ab-bee0-7301c97a6104-kube-api-access-p8q56\") pod \"auth-test-model-517a25-kserve-8458d988fc-64lmw\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") " pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:04.933338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:04.933225    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:05.258521 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:14:05.258497    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc0f79fe_2234_47ab_bee0_7301c97a6104.slice/crio-8531e157c4da3ecf78d01436617bbc52789edd115ef9192177fe94cbbbc8e759 WatchSource:0}: Error finding container 8531e157c4da3ecf78d01436617bbc52789edd115ef9192177fe94cbbbc8e759: Status 404 returned error can't find the container with id 8531e157c4da3ecf78d01436617bbc52789edd115ef9192177fe94cbbbc8e759
Oct 30 13:14:05.259617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:05.259594    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"]
Oct 30 13:14:05.688941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:05.688906    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw" event={"ID":"fc0f79fe-2234-47ab-bee0-7301c97a6104","Type":"ContainerStarted","Data":"7131a3b837859cbd39221424f762be3aa494d53d1d1ca0e7fe1bcfdc306d6743"}
Oct 30 13:14:05.689336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:05.688955    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw" event={"ID":"fc0f79fe-2234-47ab-bee0-7301c97a6104","Type":"ContainerStarted","Data":"8531e157c4da3ecf78d01436617bbc52789edd115ef9192177fe94cbbbc8e759"}
Oct 30 13:14:05.736519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:05.736467    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw" podStartSLOduration=1.7364517369999999 podStartE2EDuration="1.736451737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:05.735755038 +0000 UTC m=+2503.791248880" watchObservedRunningTime="2026-10-30 13:14:05.736451737 +0000 UTC m=+2503.791945562"
Oct 30 13:14:05.934232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:05.934206    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:12.387493 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:14:12.387444    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:14:14.933343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:14.933310    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:14.949685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:14.949655    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:15.791083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:15.791054    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:14:42.372812 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:14:42.372770    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:14:58.030801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.030764    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"]
Oct 30 13:14:58.038008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.037980    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.039968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.039939    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"]
Oct 30 13:14:58.056788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.056766    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-241ffa\"/\"kube-root-ca.crt\""
Oct 30 13:14:58.070749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.070726    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-241ffa\"/\"default-dockercfg-rjfgf\""
Oct 30 13:14:58.119696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.119671    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-241ffa\"/\"auth-test-model-241ffa-kserve-self-signed-certs\""
Oct 30 13:14:58.149652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.149628    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-home\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.149805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.149662    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-model-cache\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.149805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.149775    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9a7b93-aaf2-4ae4-b353-168065918adf-tls-certs\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.149909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.149878    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnjr\" (UniqueName: \"kubernetes.io/projected/7a9a7b93-aaf2-4ae4-b353-168065918adf-kube-api-access-spnjr\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.149958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.149927    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-tmp-dir\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.150010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.149991    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-dshm\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.159252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.159230    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-241ffa\"/\"openshift-service-ca.crt\""
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.272666    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-home\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.272707    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-model-cache\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.272750    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9a7b93-aaf2-4ae4-b353-168065918adf-tls-certs\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.272835    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnjr\" (UniqueName: \"kubernetes.io/projected/7a9a7b93-aaf2-4ae4-b353-168065918adf-kube-api-access-spnjr\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.272890    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-tmp-dir\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.272911    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-dshm\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.273871    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-home\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.274149    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-model-cache\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.276307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.274227    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-tmp-dir\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.295228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.279381    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-dshm\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.295228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.284009    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9a7b93-aaf2-4ae4-b353-168065918adf-tls-certs\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.300081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.300042    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnjr\" (UniqueName: \"kubernetes.io/projected/7a9a7b93-aaf2-4ae4-b353-168065918adf-kube-api-access-spnjr\") pod \"auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.353811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.353780    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:14:58.708145 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:14:58.708119    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a9a7b93_aaf2_4ae4_b353_168065918adf.slice/crio-25b887f05ee787ebb8fbce23fd064fb62bc97dc1171c1178666ab08ce22d6fa3 WatchSource:0}: Error finding container 25b887f05ee787ebb8fbce23fd064fb62bc97dc1171c1178666ab08ce22d6fa3: Status 404 returned error can't find the container with id 25b887f05ee787ebb8fbce23fd064fb62bc97dc1171c1178666ab08ce22d6fa3
Oct 30 13:14:58.709434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:58.709412    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"]
Oct 30 13:14:59.083397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:59.083314    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8" event={"ID":"7a9a7b93-aaf2-4ae4-b353-168065918adf","Type":"ContainerStarted","Data":"2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a"}
Oct 30 13:14:59.083397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:59.083364    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8" event={"ID":"7a9a7b93-aaf2-4ae4-b353-168065918adf","Type":"ContainerStarted","Data":"25b887f05ee787ebb8fbce23fd064fb62bc97dc1171c1178666ab08ce22d6fa3"}
Oct 30 13:14:59.133209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:59.133159    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8" podStartSLOduration=2.133144334 podStartE2EDuration="2.133144334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:59.128701 +0000 UTC m=+2557.184194823" watchObservedRunningTime="2026-10-30 13:14:59.133144334 +0000 UTC m=+2557.188638162"
Oct 30 13:14:59.354498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:14:59.354425    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:15:00.213811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.213781    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"]
Oct 30 13:15:00.218374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.218355    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:00.275725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.275689    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"]
Oct 30 13:15:00.280874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.280396    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"]
Oct 30 13:15:00.286809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.286781    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:00.297936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.297436    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:15:00.304180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.304148    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7784\" (UniqueName: \"kubernetes.io/projected/1248dd41-cadc-48c7-840d-32f7fc296f6c-kube-api-access-v7784\") pod \"maas-api-key-cleanup-29889435-hlrwf\" (UID: \"1248dd41-cadc-48c7-840d-32f7fc296f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:00.304583 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.304560    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1248dd41-cadc-48c7-840d-32f7fc296f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-hlrwf\" (UID: \"1248dd41-cadc-48c7-840d-32f7fc296f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:00.309125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.309103    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"]
Oct 30 13:15:00.406085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.406048    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db73b6-604c-4d87-b93a-e03a2f68e0ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq\" (UID: \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:00.406309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.406247    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8kt\" (UniqueName: \"kubernetes.io/projected/86db73b6-604c-4d87-b93a-e03a2f68e0ee-kube-api-access-9z8kt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq\" (UID: \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:00.406393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.406317    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1248dd41-cadc-48c7-840d-32f7fc296f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-hlrwf\" (UID: \"1248dd41-cadc-48c7-840d-32f7fc296f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:00.406614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.406389    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7784\" (UniqueName: \"kubernetes.io/projected/1248dd41-cadc-48c7-840d-32f7fc296f6c-kube-api-access-v7784\") pod \"maas-api-key-cleanup-29889435-hlrwf\" (UID: \"1248dd41-cadc-48c7-840d-32f7fc296f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:00.406996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.406974    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1248dd41-cadc-48c7-840d-32f7fc296f6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-hlrwf\" (UID: \"1248dd41-cadc-48c7-840d-32f7fc296f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:00.435725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.435699    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7784\" (UniqueName: \"kubernetes.io/projected/1248dd41-cadc-48c7-840d-32f7fc296f6c-kube-api-access-v7784\") pod \"maas-api-key-cleanup-29889435-hlrwf\" (UID: \"1248dd41-cadc-48c7-840d-32f7fc296f6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:00.508231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.508138    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db73b6-604c-4d87-b93a-e03a2f68e0ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq\" (UID: \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:00.508456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.508362    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8kt\" (UniqueName: \"kubernetes.io/projected/86db73b6-604c-4d87-b93a-e03a2f68e0ee-kube-api-access-9z8kt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq\" (UID: \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:00.508851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.508830    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db73b6-604c-4d87-b93a-e03a2f68e0ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq\" (UID: \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:00.533777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.533754    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:00.538474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.538450    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8kt\" (UniqueName: \"kubernetes.io/projected/86db73b6-604c-4d87-b93a-e03a2f68e0ee-kube-api-access-9z8kt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq\" (UID: \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:00.613617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.613587    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:00.841086 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:15:00.841052    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1248dd41_cadc_48c7_840d_32f7fc296f6c.slice/crio-1edf6af962a1e8d6124e033c0720bac11cdebeab164c31b51fd4306728d8060d WatchSource:0}: Error finding container 1edf6af962a1e8d6124e033c0720bac11cdebeab164c31b51fd4306728d8060d: Status 404 returned error can't find the container with id 1edf6af962a1e8d6124e033c0720bac11cdebeab164c31b51fd4306728d8060d
Oct 30 13:15:00.842167 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.842144    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"]
Oct 30 13:15:00.947211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:00.947180    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"]
Oct 30 13:15:00.947377 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:15:00.947268    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86db73b6_604c_4d87_b93a_e03a2f68e0ee.slice/crio-026b786d077b1ff737457eb184afb73a8f0579eae7245b6f943bf918967fb9af WatchSource:0}: Error finding container 026b786d077b1ff737457eb184afb73a8f0579eae7245b6f943bf918967fb9af: Status 404 returned error can't find the container with id 026b786d077b1ff737457eb184afb73a8f0579eae7245b6f943bf918967fb9af
Oct 30 13:15:01.097681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:01.097589    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq" event={"ID":"86db73b6-604c-4d87-b93a-e03a2f68e0ee","Type":"ContainerStarted","Data":"026b786d077b1ff737457eb184afb73a8f0579eae7245b6f943bf918967fb9af"}
Oct 30 13:15:01.099014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:01.098990    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf" event={"ID":"1248dd41-cadc-48c7-840d-32f7fc296f6c","Type":"ContainerStarted","Data":"1edf6af962a1e8d6124e033c0720bac11cdebeab164c31b51fd4306728d8060d"}
Oct 30 13:15:02.108104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:02.108015    4429 generic.go:356] "Generic (PLEG): container finished" podID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerID="e19842ad9915ab233d93942159960af40a45896c59ce1466b1d38232a6488492" exitCode=60
Oct 30 13:15:02.108556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:02.108102    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq" event={"ID":"86db73b6-604c-4d87-b93a-e03a2f68e0ee","Type":"ContainerDied","Data":"e19842ad9915ab233d93942159960af40a45896c59ce1466b1d38232a6488492"}
Oct 30 13:15:02.108556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:02.108388    4429 scope.go:122] "RemoveContainer" containerID="e19842ad9915ab233d93942159960af40a45896c59ce1466b1d38232a6488492"
Oct 30 13:15:02.110235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:02.110133    4429 generic.go:356] "Generic (PLEG): container finished" podID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerID="e8361588082136d5e87bdc744de430d599e59534375421af3417f13f01917a78" exitCode=60
Oct 30 13:15:02.110235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:02.110198    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf" event={"ID":"1248dd41-cadc-48c7-840d-32f7fc296f6c","Type":"ContainerDied","Data":"e8361588082136d5e87bdc744de430d599e59534375421af3417f13f01917a78"}
Oct 30 13:15:02.110509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:02.110493    4429 scope.go:122] "RemoveContainer" containerID="e8361588082136d5e87bdc744de430d599e59534375421af3417f13f01917a78"
Oct 30 13:15:03.119620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:03.119595    4429 generic.go:356] "Generic (PLEG): container finished" podID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerID="30e22b4131b4813cdb443102fe2c3efff53a389b8e5ef79d6160086dbb36aa64" exitCode=60
Oct 30 13:15:03.119990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:03.119669    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq" event={"ID":"86db73b6-604c-4d87-b93a-e03a2f68e0ee","Type":"ContainerDied","Data":"30e22b4131b4813cdb443102fe2c3efff53a389b8e5ef79d6160086dbb36aa64"}
Oct 30 13:15:03.119990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:03.119734    4429 scope.go:122] "RemoveContainer" containerID="e19842ad9915ab233d93942159960af40a45896c59ce1466b1d38232a6488492"
Oct 30 13:15:03.120099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:03.120010    4429 scope.go:122] "RemoveContainer" containerID="30e22b4131b4813cdb443102fe2c3efff53a389b8e5ef79d6160086dbb36aa64"
Oct 30 13:15:03.120462 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:15:03.120433    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq_odh-ai-gateway-infra(86db73b6-604c-4d87-b93a-e03a2f68e0ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee"
Oct 30 13:15:03.121981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:03.121963    4429 generic.go:356] "Generic (PLEG): container finished" podID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerID="1355d2abb74a0f950feff5a31cac48ccbf9d28d3e09674384942ef533d39ac8e" exitCode=60
Oct 30 13:15:03.122059 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:03.121992    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf" event={"ID":"1248dd41-cadc-48c7-840d-32f7fc296f6c","Type":"ContainerDied","Data":"1355d2abb74a0f950feff5a31cac48ccbf9d28d3e09674384942ef533d39ac8e"}
Oct 30 13:15:03.122313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:03.122297    4429 scope.go:122] "RemoveContainer" containerID="1355d2abb74a0f950feff5a31cac48ccbf9d28d3e09674384942ef533d39ac8e"
Oct 30 13:15:03.122625 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:15:03.122608    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-hlrwf_odh-ai-gateway-infra(1248dd41-cadc-48c7-840d-32f7fc296f6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c"
Oct 30 13:15:03.155849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:03.155829    4429 scope.go:122] "RemoveContainer" containerID="e8361588082136d5e87bdc744de430d599e59534375421af3417f13f01917a78"
Oct 30 13:15:04.130314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:04.130251    4429 scope.go:122] "RemoveContainer" containerID="1355d2abb74a0f950feff5a31cac48ccbf9d28d3e09674384942ef533d39ac8e"
Oct 30 13:15:04.130770 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:15:04.130643    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-hlrwf_odh-ai-gateway-infra(1248dd41-cadc-48c7-840d-32f7fc296f6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c"
Oct 30 13:15:04.132181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:04.132161    4429 scope.go:122] "RemoveContainer" containerID="30e22b4131b4813cdb443102fe2c3efff53a389b8e5ef79d6160086dbb36aa64"
Oct 30 13:15:04.132503 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:15:04.132482    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq_odh-ai-gateway-infra(86db73b6-604c-4d87-b93a-e03a2f68e0ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee"
Oct 30 13:15:08.354916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:08.354539    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:15:08.369097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:08.369067    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:15:09.185268 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.185244    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:15:09.684906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.684880    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"]
Oct 30 13:15:09.690760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.690739    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:09.708673 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.708651    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:09.714951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.714918    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"]
Oct 30 13:15:09.875352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.875317    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086891b8-e893-4758-861e-5795d958f7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:09.875559 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.875388    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086891b8-e893-4758-861e-5795d958f7ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:09.875628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.875609    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzk4m\" (UniqueName: \"kubernetes.io/projected/086891b8-e893-4758-861e-5795d958f7ec-kube-api-access-hzk4m\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:09.977108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.977031    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086891b8-e893-4758-861e-5795d958f7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:09.977108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.977099    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086891b8-e893-4758-861e-5795d958f7ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:09.977331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.977263    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzk4m\" (UniqueName: \"kubernetes.io/projected/086891b8-e893-4758-861e-5795d958f7ec-kube-api-access-hzk4m\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:09.977837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.977811    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086891b8-e893-4758-861e-5795d958f7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:09.979693 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:09.979663    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086891b8-e893-4758-861e-5795d958f7ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:10.007150 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:10.007125    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzk4m\" (UniqueName: \"kubernetes.io/projected/086891b8-e893-4758-861e-5795d958f7ec-kube-api-access-hzk4m\") pod \"maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:10.010043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:10.010016    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:10.322337 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:15:10.322300    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086891b8_e893_4758_861e_5795d958f7ec.slice/crio-e2cbca602ac24e7e40a4c0e20a249bb036ccac2248fb9e961689ffc1f785c680 WatchSource:0}: Error finding container e2cbca602ac24e7e40a4c0e20a249bb036ccac2248fb9e961689ffc1f785c680: Status 404 returned error can't find the container with id e2cbca602ac24e7e40a4c0e20a249bb036ccac2248fb9e961689ffc1f785c680
Oct 30 13:15:10.324700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:10.324670    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"]
Oct 30 13:15:11.192266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:11.192228    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj" event={"ID":"086891b8-e893-4758-861e-5795d958f7ec","Type":"ContainerStarted","Data":"fb3a4783813170092a42730a22c14efd83b193b728e3e4de7508571361a72756"}
Oct 30 13:15:11.192768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:11.192306    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj" event={"ID":"086891b8-e893-4758-861e-5795d958f7ec","Type":"ContainerStarted","Data":"e2cbca602ac24e7e40a4c0e20a249bb036ccac2248fb9e961689ffc1f785c680"}
Oct 30 13:15:12.201308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:12.201246    4429 generic.go:356] "Generic (PLEG): container finished" podID="086891b8-e893-4758-861e-5795d958f7ec" containerID="fb3a4783813170092a42730a22c14efd83b193b728e3e4de7508571361a72756" exitCode=0
Oct 30 13:15:12.201707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:12.201343    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj" event={"ID":"086891b8-e893-4758-861e-5795d958f7ec","Type":"ContainerDied","Data":"fb3a4783813170092a42730a22c14efd83b193b728e3e4de7508571361a72756"}
Oct 30 13:15:12.372129 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:15:12.372088    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:15:13.371449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.371426    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:13.437693 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.437657    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086891b8-e893-4758-861e-5795d958f7ec-kube-api-access-hzk4m\" (UniqueName: \"kubernetes.io/projected/086891b8-e893-4758-861e-5795d958f7ec-kube-api-access-hzk4m\") pod \"086891b8-e893-4758-861e-5795d958f7ec\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") "
Oct 30 13:15:13.437867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.437746    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086891b8-e893-4758-861e-5795d958f7ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086891b8-e893-4758-861e-5795d958f7ec-maas-api-xks-ca\") pod \"086891b8-e893-4758-861e-5795d958f7ec\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") "
Oct 30 13:15:13.438009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.437986    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086891b8-e893-4758-861e-5795d958f7ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086891b8-e893-4758-861e-5795d958f7ec-maas-api-openshift-ca\") pod \"086891b8-e893-4758-861e-5795d958f7ec\" (UID: \"086891b8-e893-4758-861e-5795d958f7ec\") "
Oct 30 13:15:13.438404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.438331    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086891b8-e893-4758-861e-5795d958f7ec-maas-api-openshift-ca" pod "086891b8-e893-4758-861e-5795d958f7ec" (UID: "086891b8-e893-4758-861e-5795d958f7ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:13.438937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.438915    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086891b8-e893-4758-861e-5795d958f7ec-maas-api-openshift-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.440088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.440068    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086891b8-e893-4758-861e-5795d958f7ec-kube-api-access-hzk4m" pod "086891b8-e893-4758-861e-5795d958f7ec" (UID: "086891b8-e893-4758-861e-5795d958f7ec"). InnerVolumeSpecName "kube-api-access-hzk4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:13.440406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.440378    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086891b8-e893-4758-861e-5795d958f7ec-maas-api-xks-ca" pod "086891b8-e893-4758-861e-5795d958f7ec" (UID: "086891b8-e893-4758-861e-5795d958f7ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:13.540004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.539932    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzk4m\" (UniqueName: \"kubernetes.io/projected/086891b8-e893-4758-861e-5795d958f7ec-kube-api-access-hzk4m\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.540004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:13.539957    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086891b8-e893-4758-861e-5795d958f7ec-maas-api-xks-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:14.077725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:14.077696    4429 scope.go:122] "RemoveContainer" containerID="1355d2abb74a0f950feff5a31cac48ccbf9d28d3e09674384942ef533d39ac8e"
Oct 30 13:15:14.215986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:14.215966    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"
Oct 30 13:15:14.216128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:14.216004    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj" event={"ID":"086891b8-e893-4758-861e-5795d958f7ec","Type":"ContainerDied","Data":"e2cbca602ac24e7e40a4c0e20a249bb036ccac2248fb9e961689ffc1f785c680"}
Oct 30 13:15:14.216128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:14.216086    4429 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2cbca602ac24e7e40a4c0e20a249bb036ccac2248fb9e961689ffc1f785c680"
Oct 30 13:15:15.226051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:15.226022    4429 generic.go:356] "Generic (PLEG): container finished" podID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerID="3b829fbfc234cadd439319ac7ecce6d9b7e26b15498d854a47912485473584d9" exitCode=60
Oct 30 13:15:15.226537 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:15.226058    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf" event={"ID":"1248dd41-cadc-48c7-840d-32f7fc296f6c","Type":"ContainerDied","Data":"3b829fbfc234cadd439319ac7ecce6d9b7e26b15498d854a47912485473584d9"}
Oct 30 13:15:15.226537 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:15.226134    4429 scope.go:122] "RemoveContainer" containerID="1355d2abb74a0f950feff5a31cac48ccbf9d28d3e09674384942ef533d39ac8e"
Oct 30 13:15:15.226679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:15.226554    4429 scope.go:122] "RemoveContainer" containerID="3b829fbfc234cadd439319ac7ecce6d9b7e26b15498d854a47912485473584d9"
Oct 30 13:15:15.226903 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:15:15.226874    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-hlrwf_odh-ai-gateway-infra(1248dd41-cadc-48c7-840d-32f7fc296f6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c"
Oct 30 13:15:16.289886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:16.289848    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"]
Oct 30 13:15:16.452863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:16.452837    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:16.582507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:16.582429    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1248dd41-cadc-48c7-840d-32f7fc296f6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1248dd41-cadc-48c7-840d-32f7fc296f6c-maas-api-service-ca\") pod \"1248dd41-cadc-48c7-840d-32f7fc296f6c\" (UID: \"1248dd41-cadc-48c7-840d-32f7fc296f6c\") "
Oct 30 13:15:16.582684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:16.582666    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1248dd41-cadc-48c7-840d-32f7fc296f6c-kube-api-access-v7784\" (UniqueName: \"kubernetes.io/projected/1248dd41-cadc-48c7-840d-32f7fc296f6c-kube-api-access-v7784\") pod \"1248dd41-cadc-48c7-840d-32f7fc296f6c\" (UID: \"1248dd41-cadc-48c7-840d-32f7fc296f6c\") "
Oct 30 13:15:16.582791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:16.582754    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1248dd41-cadc-48c7-840d-32f7fc296f6c-maas-api-service-ca" pod "1248dd41-cadc-48c7-840d-32f7fc296f6c" (UID: "1248dd41-cadc-48c7-840d-32f7fc296f6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:16.583060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:16.583045    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1248dd41-cadc-48c7-840d-32f7fc296f6c-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:16.584880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:16.584860    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1248dd41-cadc-48c7-840d-32f7fc296f6c-kube-api-access-v7784" pod "1248dd41-cadc-48c7-840d-32f7fc296f6c" (UID: "1248dd41-cadc-48c7-840d-32f7fc296f6c"). InnerVolumeSpecName "kube-api-access-v7784". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:16.685378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:16.685350    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7784\" (UniqueName: \"kubernetes.io/projected/1248dd41-cadc-48c7-840d-32f7fc296f6c-kube-api-access-v7784\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:17.067900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:17.067866    4429 scope.go:122] "RemoveContainer" containerID="30e22b4131b4813cdb443102fe2c3efff53a389b8e5ef79d6160086dbb36aa64"
Oct 30 13:15:17.243820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:17.243788    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"
Oct 30 13:15:17.244005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:17.243789    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf" event={"ID":"1248dd41-cadc-48c7-840d-32f7fc296f6c","Type":"ContainerDied","Data":"1edf6af962a1e8d6124e033c0720bac11cdebeab164c31b51fd4306728d8060d"}
Oct 30 13:15:17.244005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:17.243944    4429 scope.go:122] "RemoveContainer" containerID="3b829fbfc234cadd439319ac7ecce6d9b7e26b15498d854a47912485473584d9"
Oct 30 13:15:17.333447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:17.333356    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"]
Oct 30 13:15:17.356098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:17.349350    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-hlrwf"]
Oct 30 13:15:18.076532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:18.076504    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" path="/var/lib/kubelet/pods/1248dd41-cadc-48c7-840d-32f7fc296f6c/volumes"
Oct 30 13:15:18.252106 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:18.252076    4429 generic.go:356] "Generic (PLEG): container finished" podID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerID="1274370607bff404895993b873f03862a574c6e0a4644959bad56b3711f770de" exitCode=60
Oct 30 13:15:18.252312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:18.252198    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq" event={"ID":"86db73b6-604c-4d87-b93a-e03a2f68e0ee","Type":"ContainerDied","Data":"1274370607bff404895993b873f03862a574c6e0a4644959bad56b3711f770de"}
Oct 30 13:15:18.252312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:18.252256    4429 scope.go:122] "RemoveContainer" containerID="30e22b4131b4813cdb443102fe2c3efff53a389b8e5ef79d6160086dbb36aa64"
Oct 30 13:15:18.252810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:18.252787    4429 scope.go:122] "RemoveContainer" containerID="1274370607bff404895993b873f03862a574c6e0a4644959bad56b3711f770de"
Oct 30 13:15:18.253195 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:15:18.253168    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq_odh-ai-gateway-infra(86db73b6-604c-4d87-b93a-e03a2f68e0ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee"
Oct 30 13:15:19.311559 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:19.311520    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"]
Oct 30 13:15:19.481154 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:19.481126    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:19.639883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:19.639794    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86db73b6-604c-4d87-b93a-e03a2f68e0ee-kube-api-access-9z8kt\" (UniqueName: \"kubernetes.io/projected/86db73b6-604c-4d87-b93a-e03a2f68e0ee-kube-api-access-9z8kt\") pod \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\" (UID: \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\") "
Oct 30 13:15:19.639883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:19.639838    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86db73b6-604c-4d87-b93a-e03a2f68e0ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db73b6-604c-4d87-b93a-e03a2f68e0ee-maas-api-service-ca\") pod \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\" (UID: \"86db73b6-604c-4d87-b93a-e03a2f68e0ee\") "
Oct 30 13:15:19.640314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:19.640267    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86db73b6-604c-4d87-b93a-e03a2f68e0ee-maas-api-service-ca" pod "86db73b6-604c-4d87-b93a-e03a2f68e0ee" (UID: "86db73b6-604c-4d87-b93a-e03a2f68e0ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:19.640653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:19.640637    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db73b6-604c-4d87-b93a-e03a2f68e0ee-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:19.642187 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:19.642163    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86db73b6-604c-4d87-b93a-e03a2f68e0ee-kube-api-access-9z8kt" pod "86db73b6-604c-4d87-b93a-e03a2f68e0ee" (UID: "86db73b6-604c-4d87-b93a-e03a2f68e0ee"). InnerVolumeSpecName "kube-api-access-9z8kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:19.742079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:19.742041    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8kt\" (UniqueName: \"kubernetes.io/projected/86db73b6-604c-4d87-b93a-e03a2f68e0ee-kube-api-access-9z8kt\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.225937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:20.225898    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"]
Oct 30 13:15:20.268123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:20.268093    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"]
Oct 30 13:15:20.272635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:20.272596    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"
Oct 30 13:15:20.272850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:20.272604    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq" event={"ID":"86db73b6-604c-4d87-b93a-e03a2f68e0ee","Type":"ContainerDied","Data":"026b786d077b1ff737457eb184afb73a8f0579eae7245b6f943bf918967fb9af"}
Oct 30 13:15:20.272850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:20.272783    4429 scope.go:122] "RemoveContainer" containerID="1274370607bff404895993b873f03862a574c6e0a4644959bad56b3711f770de"
Oct 30 13:15:20.395325 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:20.395213    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"]
Oct 30 13:15:20.401020 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:20.400987    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71fbeb-29889435-mjmjq"]
Oct 30 13:15:22.080327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:22.080273    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" path="/var/lib/kubelet/pods/86db73b6-604c-4d87-b93a-e03a2f68e0ee/volumes"
Oct 30 13:15:25.226891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:25.226832    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx" podUID="aa257796-b27a-4909-abd7-d99506b2addb" containerName="payload-processing" containerID="cri-o://d0656018a3b5266597511083a5e7fe238721ce6e8006406b7369fee83992e2f3" gracePeriod=25
Oct 30 13:15:25.268710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:25.268662    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj" podUID="1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" containerName="payload-pre-processing" containerID="cri-o://953d6dbbd4e998752d37eb8a6c16999ebf8e5669fe55b73f74ddf5d2aae55630" gracePeriod=25
Oct 30 13:15:31.347431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:31.347388    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx" podUID="aa257796-b27a-4909-abd7-d99506b2addb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:15:31.347431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:31.347425    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj" podUID="1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:15:41.347370 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:41.347335    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx" podUID="aa257796-b27a-4909-abd7-d99506b2addb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:15:41.347370 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:41.347364    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj" podUID="1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:15:42.372899 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:15:42.372861    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:15:45.473057 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.473026    4429 generic.go:356] "Generic (PLEG): container finished" podID="aa257796-b27a-4909-abd7-d99506b2addb" containerID="d0656018a3b5266597511083a5e7fe238721ce6e8006406b7369fee83992e2f3" exitCode=0
Oct 30 13:15:45.473644 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.473349    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx" event={"ID":"aa257796-b27a-4909-abd7-d99506b2addb","Type":"ContainerDied","Data":"d0656018a3b5266597511083a5e7fe238721ce6e8006406b7369fee83992e2f3"}
Oct 30 13:15:45.475507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.475305    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" containerID="953d6dbbd4e998752d37eb8a6c16999ebf8e5669fe55b73f74ddf5d2aae55630" exitCode=0
Oct 30 13:15:45.475507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.475463    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj" event={"ID":"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc","Type":"ContainerDied","Data":"953d6dbbd4e998752d37eb8a6c16999ebf8e5669fe55b73f74ddf5d2aae55630"}
Oct 30 13:15:45.565500 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.565474    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:15:45.593357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.593336    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:15:45.666739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.666684    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa257796-b27a-4909-abd7-d99506b2addb-kube-api-access-9k55r\" (UniqueName: \"kubernetes.io/projected/aa257796-b27a-4909-abd7-d99506b2addb-kube-api-access-9k55r\") pod \"aa257796-b27a-4909-abd7-d99506b2addb\" (UID: \"aa257796-b27a-4909-abd7-d99506b2addb\") "
Oct 30 13:15:45.666959 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.666807    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa257796-b27a-4909-abd7-d99506b2addb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa257796-b27a-4909-abd7-d99506b2addb-plugins-config-volume\") pod \"aa257796-b27a-4909-abd7-d99506b2addb\" (UID: \"aa257796-b27a-4909-abd7-d99506b2addb\") "
Oct 30 13:15:45.667381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.667351    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa257796-b27a-4909-abd7-d99506b2addb-plugins-config-volume" pod "aa257796-b27a-4909-abd7-d99506b2addb" (UID: "aa257796-b27a-4909-abd7-d99506b2addb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:45.667530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.667505    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa257796-b27a-4909-abd7-d99506b2addb-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:45.669745 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.669717    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa257796-b27a-4909-abd7-d99506b2addb-kube-api-access-9k55r" pod "aa257796-b27a-4909-abd7-d99506b2addb" (UID: "aa257796-b27a-4909-abd7-d99506b2addb"). InnerVolumeSpecName "kube-api-access-9k55r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:45.769086 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.769054    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-kube-api-access-wsw8n\" (UniqueName: \"kubernetes.io/projected/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-kube-api-access-wsw8n\") pod \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\" (UID: \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\") "
Oct 30 13:15:45.769329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.769147    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-plugins-config-volume\") pod \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\" (UID: \"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc\") "
Oct 30 13:15:45.769618 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.769593    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-plugins-config-volume" pod "1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" (UID: "1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:45.769955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.769935    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:45.770004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.769960    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k55r\" (UniqueName: \"kubernetes.io/projected/aa257796-b27a-4909-abd7-d99506b2addb-kube-api-access-9k55r\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:45.771387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.771365    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-kube-api-access-wsw8n" pod "1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" (UID: "1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc"). InnerVolumeSpecName "kube-api-access-wsw8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:45.871195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:45.871110    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsw8n\" (UniqueName: \"kubernetes.io/projected/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc-kube-api-access-wsw8n\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:15:46.483235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.483199    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"
Oct 30 13:15:46.483744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.483202    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx" event={"ID":"aa257796-b27a-4909-abd7-d99506b2addb","Type":"ContainerDied","Data":"ee06cfe10e6a113f281bb87761b55e7f5924a63ed6295a309453bc62f8f18e8c"}
Oct 30 13:15:46.483744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.483369    4429 scope.go:122] "RemoveContainer" containerID="d0656018a3b5266597511083a5e7fe238721ce6e8006406b7369fee83992e2f3"
Oct 30 13:15:46.484950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.484927    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"
Oct 30 13:15:46.485063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.484961    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj" event={"ID":"1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc","Type":"ContainerDied","Data":"8fe04bd13d9ea7f38a7eb9e41fd2e3d73872c58c0b8d137ed5d1b5df5cf60dc2"}
Oct 30 13:15:46.493812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.493789    4429 scope.go:122] "RemoveContainer" containerID="953d6dbbd4e998752d37eb8a6c16999ebf8e5669fe55b73f74ddf5d2aae55630"
Oct 30 13:15:46.543354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.543312    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"]
Oct 30 13:15:46.559442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.559408    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2fdf91-7bf7c4c4f9-wtbdj"]
Oct 30 13:15:46.599347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.599318    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"]
Oct 30 13:15:46.604748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:46.604198    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2fdf91-6d85599cb8-dntxx"]
Oct 30 13:15:48.077303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:48.077253    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" path="/var/lib/kubelet/pods/1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc/volumes"
Oct 30 13:15:48.077742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:48.077726    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa257796-b27a-4909-abd7-d99506b2addb" path="/var/lib/kubelet/pods/aa257796-b27a-4909-abd7-d99506b2addb/volumes"
Oct 30 13:15:59.077574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:59.077543    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"]
Oct 30 13:15:59.090812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:15:59.090788    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2fdf91-s2mzj"]
Oct 30 13:16:00.077166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:00.077129    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086891b8-e893-4758-861e-5795d958f7ec" path="/var/lib/kubelet/pods/086891b8-e893-4758-861e-5795d958f7ec/volumes"
Oct 30 13:16:04.630318 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:04.630274    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"]
Oct 30 13:16:12.373761 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:16:12.373711    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:16:19.652916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:19.652842    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv" podUID="0116a939-fad3-43ad-8e7a-4d4ec958c685" containerName="main" containerID="cri-o://f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929" gracePeriod=45
Oct 30 13:16:19.940717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:19.940690    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:16:20.084655 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.084629    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-tmp-dir\") pod \"0116a939-fad3-43ad-8e7a-4d4ec958c685\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") "
Oct 30 13:16:20.084812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.084765    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-home\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-home\") pod \"0116a939-fad3-43ad-8e7a-4d4ec958c685\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") "
Oct 30 13:16:20.084868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.084810    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0116a939-fad3-43ad-8e7a-4d4ec958c685-kube-api-access-kk5l9\" (UniqueName: \"kubernetes.io/projected/0116a939-fad3-43ad-8e7a-4d4ec958c685-kube-api-access-kk5l9\") pod \"0116a939-fad3-43ad-8e7a-4d4ec958c685\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") "
Oct 30 13:16:20.084909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.084860    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-tmp-dir" pod "0116a939-fad3-43ad-8e7a-4d4ec958c685" (UID: "0116a939-fad3-43ad-8e7a-4d4ec958c685"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:20.084960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.084910    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0116a939-fad3-43ad-8e7a-4d4ec958c685-tls-certs\" (UniqueName: \"kubernetes.io/secret/0116a939-fad3-43ad-8e7a-4d4ec958c685-tls-certs\") pod \"0116a939-fad3-43ad-8e7a-4d4ec958c685\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") "
Oct 30 13:16:20.084960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.084955    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-dshm\") pod \"0116a939-fad3-43ad-8e7a-4d4ec958c685\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") "
Oct 30 13:16:20.085036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.084997    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-model-cache\") pod \"0116a939-fad3-43ad-8e7a-4d4ec958c685\" (UID: \"0116a939-fad3-43ad-8e7a-4d4ec958c685\") "
Oct 30 13:16:20.085073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.085038    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-home" pod "0116a939-fad3-43ad-8e7a-4d4ec958c685" (UID: "0116a939-fad3-43ad-8e7a-4d4ec958c685"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:20.085345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.085320    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-model-cache" pod "0116a939-fad3-43ad-8e7a-4d4ec958c685" (UID: "0116a939-fad3-43ad-8e7a-4d4ec958c685"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:20.085630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.085510    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:16:20.085630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.085525    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:16:20.085630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.085538    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:16:20.087238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.087209    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0116a939-fad3-43ad-8e7a-4d4ec958c685-kube-api-access-kk5l9" pod "0116a939-fad3-43ad-8e7a-4d4ec958c685" (UID: "0116a939-fad3-43ad-8e7a-4d4ec958c685"). InnerVolumeSpecName "kube-api-access-kk5l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:20.087238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.087210    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-dshm" pod "0116a939-fad3-43ad-8e7a-4d4ec958c685" (UID: "0116a939-fad3-43ad-8e7a-4d4ec958c685"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:20.087389 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.087215    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0116a939-fad3-43ad-8e7a-4d4ec958c685-tls-certs" pod "0116a939-fad3-43ad-8e7a-4d4ec958c685" (UID: "0116a939-fad3-43ad-8e7a-4d4ec958c685"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:20.186929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.186901    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0116a939-fad3-43ad-8e7a-4d4ec958c685-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:16:20.187090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.186930    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk5l9\" (UniqueName: \"kubernetes.io/projected/0116a939-fad3-43ad-8e7a-4d4ec958c685-kube-api-access-kk5l9\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:16:20.187090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.186947    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0116a939-fad3-43ad-8e7a-4d4ec958c685-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:16:20.776274 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.776224    4429 generic.go:356] "Generic (PLEG): container finished" podID="0116a939-fad3-43ad-8e7a-4d4ec958c685" containerID="f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929" exitCode=0
Oct 30 13:16:20.776809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.776275    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv" event={"ID":"0116a939-fad3-43ad-8e7a-4d4ec958c685","Type":"ContainerDied","Data":"f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929"}
Oct 30 13:16:20.776809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.776321    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"
Oct 30 13:16:20.776809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.776357    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv" event={"ID":"0116a939-fad3-43ad-8e7a-4d4ec958c685","Type":"ContainerDied","Data":"2d533b34a8a13f01ab4903e4907e890d90fb49d19b3b2328464a279c53ecf0e1"}
Oct 30 13:16:20.776809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.776384    4429 scope.go:122] "RemoveContainer" containerID="f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929"
Oct 30 13:16:20.788112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.788095    4429 scope.go:122] "RemoveContainer" containerID="f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929"
Oct 30 13:16:20.788419 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:16:20.788402    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929\": container with ID starting with f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929 not found: ID does not exist" containerID="f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929"
Oct 30 13:16:20.788501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.788431    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929"} err="failed to get container status \"f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929\": rpc error: code = NotFound desc = could not find container \"f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929\": container with ID starting with f4166fdf30c43032ecec5c9cd15e0d327df3b524624b3007f69afcb0e9722929 not found: ID does not exist"
Oct 30 13:16:20.835563 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.835536    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"]
Oct 30 13:16:20.848904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:20.848878    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-2fdf91/praxis-route-2fdf91-kserve-79d7b59477-q8dvv"]
Oct 30 13:16:22.077078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:16:22.077045    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0116a939-fad3-43ad-8e7a-4d4ec958c685" path="/var/lib/kubelet/pods/0116a939-fad3-43ad-8e7a-4d4ec958c685/volumes"
Oct 30 13:16:42.372100 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:16:42.372018    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:17:00.144522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:00.144488    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"]
Oct 30 13:17:05.145550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:05.145514    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h" podUID="339c4cee-4f61-48ec-a96c-a5c2316edd02" containerName="payload-processing" containerID="cri-o://101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b" gracePeriod=25
Oct 30 13:17:05.422231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:05.422199    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:17:05.478730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:05.478695    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339c4cee-4f61-48ec-a96c-a5c2316edd02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c4cee-4f61-48ec-a96c-a5c2316edd02-plugins-config-volume\") pod \"339c4cee-4f61-48ec-a96c-a5c2316edd02\" (UID: \"339c4cee-4f61-48ec-a96c-a5c2316edd02\") "
Oct 30 13:17:05.478926 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:05.478891    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339c4cee-4f61-48ec-a96c-a5c2316edd02-kube-api-access-7l66l\" (UniqueName: \"kubernetes.io/projected/339c4cee-4f61-48ec-a96c-a5c2316edd02-kube-api-access-7l66l\") pod \"339c4cee-4f61-48ec-a96c-a5c2316edd02\" (UID: \"339c4cee-4f61-48ec-a96c-a5c2316edd02\") "
Oct 30 13:17:05.479113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:05.479068    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339c4cee-4f61-48ec-a96c-a5c2316edd02-plugins-config-volume" pod "339c4cee-4f61-48ec-a96c-a5c2316edd02" (UID: "339c4cee-4f61-48ec-a96c-a5c2316edd02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:05.479846 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:05.479609    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c4cee-4f61-48ec-a96c-a5c2316edd02-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:05.481957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:05.481930    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339c4cee-4f61-48ec-a96c-a5c2316edd02-kube-api-access-7l66l" pod "339c4cee-4f61-48ec-a96c-a5c2316edd02" (UID: "339c4cee-4f61-48ec-a96c-a5c2316edd02"). InnerVolumeSpecName "kube-api-access-7l66l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:05.580781 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:05.580749    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l66l\" (UniqueName: \"kubernetes.io/projected/339c4cee-4f61-48ec-a96c-a5c2316edd02-kube-api-access-7l66l\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:06.107028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.106996    4429 generic.go:356] "Generic (PLEG): container finished" podID="339c4cee-4f61-48ec-a96c-a5c2316edd02" containerID="101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b" exitCode=0
Oct 30 13:17:06.107196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.107047    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"
Oct 30 13:17:06.107196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.107080    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h" event={"ID":"339c4cee-4f61-48ec-a96c-a5c2316edd02","Type":"ContainerDied","Data":"101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b"}
Oct 30 13:17:06.107196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.107129    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h" event={"ID":"339c4cee-4f61-48ec-a96c-a5c2316edd02","Type":"ContainerDied","Data":"b4f7c582fb5bca7f1cd5a68c56a79b1392ea24ec119ce2bfc2c30504637160d2"}
Oct 30 13:17:06.107196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.107152    4429 scope.go:122] "RemoveContainer" containerID="101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b"
Oct 30 13:17:06.117295 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.117264    4429 scope.go:122] "RemoveContainer" containerID="101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b"
Oct 30 13:17:06.117607 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:06.117589    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b\": container with ID starting with 101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b not found: ID does not exist" containerID="101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b"
Oct 30 13:17:06.117665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.117614    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b"} err="failed to get container status \"101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b\": rpc error: code = NotFound desc = could not find container \"101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b\": container with ID starting with 101ba8d3f6d11a40a13a95f74e97a8dba1659f9c0f6e978d2da0e0497d040c1b not found: ID does not exist"
Oct 30 13:17:06.165869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.165831    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"]
Oct 30 13:17:06.172915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:06.172884    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca097a-868bdd8cdd-jmz5h"]
Oct 30 13:17:08.077928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:08.077897    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339c4cee-4f61-48ec-a96c-a5c2316edd02" path="/var/lib/kubelet/pods/339c4cee-4f61-48ec-a96c-a5c2316edd02/volumes"
Oct 30 13:17:12.373192 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:12.373157    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:17:25.325037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:25.325001    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"]
Oct 30 13:17:25.325484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:25.325228    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy" containerID="cri-o://4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635" gracePeriod=30
Oct 30 13:17:27.748350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:27.748318    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"]
Oct 30 13:17:27.748897 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:27.748514    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy" containerID="cri-o://2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df" gracePeriod=30
Oct 30 13:17:27.779670 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:27.779636    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"]
Oct 30 13:17:27.780105 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:27.780056    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy" containerID="cri-o://c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3" gracePeriod=30
Oct 30 13:17:27.787480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:27.787399    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"]
Oct 30 13:17:27.787958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:27.787930    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy" containerID="cri-o://9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca" gracePeriod=30
Oct 30 13:17:27.790345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:27.790304    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"]
Oct 30 13:17:27.790806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:27.790779    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy" containerID="cri-o://79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f" gracePeriod=30
Oct 30 13:17:29.458802 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:29.458773    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"]
Oct 30 13:17:29.459260 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:29.458975    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" containerID="cri-o://dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336" gracePeriod=30
Oct 30 13:17:29.473594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:29.473565    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"]
Oct 30 13:17:29.473809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:29.473785    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy" containerID="cri-o://f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92" gracePeriod=30
Oct 30 13:17:29.477576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:29.477549    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"]
Oct 30 13:17:29.477820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:29.477790    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy" containerID="cri-o://fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae" gracePeriod=30
Oct 30 13:17:29.486935 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:29.486910    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"]
Oct 30 13:17:29.487181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:29.487153    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy" containerID="cri-o://d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a" gracePeriod=30
Oct 30 13:17:30.609241 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.609184    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 13:17:30.672471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.672434    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-token\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-token\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.672681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.672559    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-credential-socket\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.672681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.672664    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-envoy\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.672838 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.672712    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-kube-api-access-kt4fd\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-kube-api-access-kt4fd\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.672838 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.672781    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-socket\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.672838 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.672827    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-podinfo\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.673026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.672985    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istiod-ca-cert\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.673088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.673022    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-certs\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.674170 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.673435    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-data\") pod \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\" (UID: \"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9\") "
Oct 30 13:17:30.674853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.674631    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-certs" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:30.674969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.674846    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-socket" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:30.674969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.674939    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istiod-ca-cert" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:30.675421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.675381    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-credential-socket" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:30.675547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.675478    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-data" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:30.679304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.675960    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:30.679304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.675988    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:30.679304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.676001    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:30.679304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.676014    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:30.679304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.676028    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:30.679742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.679711    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-podinfo" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:30.679850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.679753    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-kube-api-access-kt4fd" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "kube-api-access-kt4fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:30.679850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.679773    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-token" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:30.696691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.696649    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-envoy" pod "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" (UID: "3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:30.777401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.777362    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:30.777401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.777399    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:30.777634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.777423    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4fd\" (UniqueName: \"kubernetes.io/projected/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-kube-api-access-kt4fd\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:30.777634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:30.777437    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:31.298215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.298181    4429 generic.go:356] "Generic (PLEG): container finished" podID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerID="4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635" exitCode=0
Oct 30 13:17:31.298449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.298233    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"
Oct 30 13:17:31.298449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.298265    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" event={"ID":"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9","Type":"ContainerDied","Data":"4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635"}
Oct 30 13:17:31.298449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.298334    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl" event={"ID":"3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9","Type":"ContainerDied","Data":"af9dc23b5f2f6b271d1a5d6aedafcbd057691ea89c1880df73f59f79585c9b30"}
Oct 30 13:17:31.298449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.298365    4429 scope.go:122] "RemoveContainer" containerID="4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635"
Oct 30 13:17:31.310132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.310116    4429 scope.go:122] "RemoveContainer" containerID="d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460"
Oct 30 13:17:31.319662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.319648    4429 scope.go:122] "RemoveContainer" containerID="4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635"
Oct 30 13:17:31.319927 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:31.319908    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635\": container with ID starting with 4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635 not found: ID does not exist" containerID="4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635"
Oct 30 13:17:31.320007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.319939    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635"} err="failed to get container status \"4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635\": rpc error: code = NotFound desc = could not find container \"4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635\": container with ID starting with 4f5aa5fc12d23a04589a66a9d1e7abc6392dcf33f988cd808449d3d5762be635 not found: ID does not exist"
Oct 30 13:17:31.320007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.319969    4429 scope.go:122] "RemoveContainer" containerID="d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460"
Oct 30 13:17:31.320220 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:31.320201    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460\": container with ID starting with d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460 not found: ID does not exist" containerID="d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460"
Oct 30 13:17:31.320303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.320228    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460"} err="failed to get container status \"d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460\": rpc error: code = NotFound desc = could not find container \"d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460\": container with ID starting with d5987058171bc6518dbb39ec9f3fd80f4e097d15abbfe1fa785c3f33e10ff460 not found: ID does not exist"
Oct 30 13:17:31.377124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.377090    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"]
Oct 30 13:17:31.390799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.390757    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-fsgtl"]
Oct 30 13:17:31.495693 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.495661    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"]
Oct 30 13:17:31.495932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:31.495908    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy" containerID="cri-o://1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f" gracePeriod=30
Oct 30 13:17:32.079981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.079941    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" path="/var/lib/kubelet/pods/3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9/volumes"
Oct 30 13:17:32.593922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.593885    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11535-cc7e99"]
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.594886    4429 cpu_manager.go:414] "containerMap: removing container" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.594910    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.594929    4429 cpu_manager.go:414] "containerMap: removing container" podUID="339c4cee-4f61-48ec-a96c-a5c2316edd02" containerName="payload-processing"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.594937    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339c4cee-4f61-48ec-a96c-a5c2316edd02" containerName="payload-processing"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.594958    4429 cpu_manager.go:414] "containerMap: removing container" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.594966    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.594978    4429 cpu_manager.go:414] "containerMap: removing container" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.594985    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595042    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595050    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595063    4429 cpu_manager.go:414] "containerMap: removing container" podUID="aa257796-b27a-4909-abd7-d99506b2addb" containerName="payload-processing"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595071    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa257796-b27a-4909-abd7-d99506b2addb" containerName="payload-processing"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595086    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595093    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595104    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595111    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595121    4429 cpu_manager.go:414] "containerMap: removing container" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595128    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595144    4429 cpu_manager.go:414] "containerMap: removing container" podUID="086891b8-e893-4758-861e-5795d958f7ec" containerName="revoke-keys"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595151    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086891b8-e893-4758-861e-5795d958f7ec" containerName="revoke-keys"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595162    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" containerName="payload-pre-processing"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595170    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" containerName="payload-pre-processing"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595200    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0116a939-fad3-43ad-8e7a-4d4ec958c685" containerName="main"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595206    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0116a939-fad3-43ad-8e7a-4d4ec958c685" containerName="main"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595382    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0116a939-fad3-43ad-8e7a-4d4ec958c685" containerName="main"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595398    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595410    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="339c4cee-4f61-48ec-a96c-a5c2316edd02" containerName="payload-processing"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595420    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595429    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bcd3ba2-d0df-4a3d-ab08-18e952fe07b9" containerName="istio-proxy"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595442    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595451    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa257796-b27a-4909-abd7-d99506b2addb" containerName="payload-processing"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595462    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595473    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1248dd41-cadc-48c7-840d-32f7fc296f6c" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595481    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595489    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="086891b8-e893-4758-861e-5795d958f7ec" containerName="revoke-keys"
Oct 30 13:17:32.596460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.595499    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec8dc83-4eb9-4c1a-a724-df8d7438a0fc" containerName="payload-pre-processing"
Oct 30 13:17:32.601892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.601871    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11535-cc7e99"
Oct 30 13:17:32.614136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.614110    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11535-cc7e99"]
Oct 30 13:17:32.623487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.623462    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ljkrn\""
Oct 30 13:17:32.714363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.714331    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb68m\" (UniqueName: \"kubernetes.io/projected/a7531b42-0d02-41db-92d7-feff6d48cec1-kube-api-access-qb68m\") pod \"test-curl-11535-cc7e99\" (UID: \"a7531b42-0d02-41db-92d7-feff6d48cec1\") " pod="opendatahub/test-curl-11535-cc7e99"
Oct 30 13:17:32.816005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.815964    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb68m\" (UniqueName: \"kubernetes.io/projected/a7531b42-0d02-41db-92d7-feff6d48cec1-kube-api-access-qb68m\") pod \"test-curl-11535-cc7e99\" (UID: \"a7531b42-0d02-41db-92d7-feff6d48cec1\") " pod="opendatahub/test-curl-11535-cc7e99"
Oct 30 13:17:32.841805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.841776    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb68m\" (UniqueName: \"kubernetes.io/projected/a7531b42-0d02-41db-92d7-feff6d48cec1-kube-api-access-qb68m\") pod \"test-curl-11535-cc7e99\" (UID: \"a7531b42-0d02-41db-92d7-feff6d48cec1\") " pod="opendatahub/test-curl-11535-cc7e99"
Oct 30 13:17:32.916900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:32.916863    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11535-cc7e99"
Oct 30 13:17:33.114244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.114211    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:17:33.164432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.164406    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 13:17:33.230786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.230721    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-kube-api-access-kn6rv\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-kube-api-access-kn6rv\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.230786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.230785    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istiod-ca-cert\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.230859    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-socket\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.230911    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istiod-ca-cert\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.230965    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-socket\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231007    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-credential-socket\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231046    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-data\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231106    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-token\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231204    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-envoy\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231268    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-kube-api-access-p5mng\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-kube-api-access-p5mng\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.231372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231340    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-token\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-token\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.232181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231389    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-podinfo\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.232181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231439    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-certs\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.232181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231505    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-data\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.232181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231539    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-credential-socket\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.232181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231641    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-envoy\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.232181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231685    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-certs\") pod \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\" (UID: \"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5\") "
Oct 30 13:17:33.232181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.231758    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-podinfo\") pod \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\" (UID: \"f7860b3b-699d-424b-8a3e-ff32cf1a6cda\") "
Oct 30 13:17:33.235356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.233827    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-credential-socket" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.235356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.233783    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-socket" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.235356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.234117    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-credential-socket" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.235356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.234376    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-data" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.235356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.234501    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istiod-ca-cert" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:33.235356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.235104    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istiod-ca-cert" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:33.235356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.235314    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-certs" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.239339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.236269    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-data" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.239339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.236444    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-certs" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.242212 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.242153    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-socket" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.247773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.247751    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-kube-api-access-kn6rv" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "kube-api-access-kn6rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:33.247927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.247880    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-token" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:33.248039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.248020    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-envoy" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.248224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.248153    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-envoy" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.248224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.248199    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-kube-api-access-p5mng" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "kube-api-access-p5mng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:33.248926 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.248902    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-podinfo" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:33.250088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.249970    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-podinfo" pod "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" (UID: "c7cf78f7-11ac-490e-bbf2-b1eb66f542b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:33.251220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.251199    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-token" pod "f7860b3b-699d-424b-8a3e-ff32cf1a6cda" (UID: "f7860b3b-699d-424b-8a3e-ff32cf1a6cda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:33.261984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.261636    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:17:33.266940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.266914    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 13:17:33.281411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.281385    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11535-cc7e99"]
Oct 30 13:17:33.285827 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:17:33.285795    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7531b42_0d02_41db_92d7_feff6d48cec1.slice/crio-dcbc701cff406d413a661e65ea8adeb7ace3bf2b58a043050229aa5855b3c8a5 WatchSource:0}: Error finding container dcbc701cff406d413a661e65ea8adeb7ace3bf2b58a043050229aa5855b3c8a5: Status 404 returned error can't find the container with id dcbc701cff406d413a661e65ea8adeb7ace3bf2b58a043050229aa5855b3c8a5
Oct 30 13:17:33.322408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.322370    4429 generic.go:356] "Generic (PLEG): container finished" podID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerID="c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3" exitCode=0
Oct 30 13:17:33.322590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.322477    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"
Oct 30 13:17:33.322590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.322508    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" event={"ID":"f55cd737-74cc-4679-8dd5-fa41d82dbc47","Type":"ContainerDied","Data":"c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3"}
Oct 30 13:17:33.322718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.322590    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk" event={"ID":"f55cd737-74cc-4679-8dd5-fa41d82dbc47","Type":"ContainerDied","Data":"5f3ead7d5534abe2e0dae9b81c29f4bc2bf42ef0a6fe9c832a447dfe265b446d"}
Oct 30 13:17:33.322718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.322622    4429 scope.go:122] "RemoveContainer" containerID="c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3"
Oct 30 13:17:33.326205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.325832    4429 generic.go:356] "Generic (PLEG): container finished" podID="cab95c54-be84-4760-ab71-1d017403a629" containerID="79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f" exitCode=0
Oct 30 13:17:33.326205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.325917    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" event={"ID":"cab95c54-be84-4760-ab71-1d017403a629","Type":"ContainerDied","Data":"79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f"}
Oct 30 13:17:33.326205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.325953    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk" event={"ID":"cab95c54-be84-4760-ab71-1d017403a629","Type":"ContainerDied","Data":"8d23007edd405abccc122c618beeab2b0ef329266c214f9542e88ae7ca3abc25"}
Oct 30 13:17:33.326205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.326085    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"
Oct 30 13:17:33.329449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.329425    4429 generic.go:356] "Generic (PLEG): container finished" podID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerID="9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca" exitCode=0
Oct 30 13:17:33.329739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.329695    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"
Oct 30 13:17:33.330040 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.329902    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" event={"ID":"f7860b3b-699d-424b-8a3e-ff32cf1a6cda","Type":"ContainerDied","Data":"9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca"}
Oct 30 13:17:33.330278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.330206    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2" event={"ID":"f7860b3b-699d-424b-8a3e-ff32cf1a6cda","Type":"ContainerDied","Data":"17ed08bbc625c83e184b442ec96aca1d779af9375c97613d43d0427140899690"}
Oct 30 13:17:33.333059 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333035    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-data\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.333204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333146    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-certs\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333196    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-credential-socket\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333228    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab95c54-be84-4760-ab71-1d017403a629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab95c54-be84-4760-ab71-1d017403a629-istiod-ca-cert\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.333504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333314    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-kube-api-access-t7v89\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-kube-api-access-t7v89\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333387    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-socket\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333413    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-token\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-token\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333441    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-envoy\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333533    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-socket\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333589    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-credential-socket\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333631    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-data\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333676    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istiod-ca-cert\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333725    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-kube-api-access-mjt98\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-kube-api-access-mjt98\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333762    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-istio-token\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-istio-token\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333834    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-podinfo\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333879    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-envoy\") pod \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\" (UID: \"f55cd737-74cc-4679-8dd5-fa41d82dbc47\") "
Oct 30 13:17:33.333911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333907    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-certs\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.334582 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.333933    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cab95c54-be84-4760-ab71-1d017403a629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab95c54-be84-4760-ab71-1d017403a629-istio-podinfo\") pod \"cab95c54-be84-4760-ab71-1d017403a629\" (UID: \"cab95c54-be84-4760-ab71-1d017403a629\") "
Oct 30 13:17:33.334665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334591    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn6rv\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-kube-api-access-kn6rv\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.334665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334615    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.334665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334629    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.334665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334642    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.334665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334655    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334668    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334681    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334693    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334703    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334718    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5mng\" (UniqueName: \"kubernetes.io/projected/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-kube-api-access-p5mng\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334731    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334743    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334755    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334767    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334778    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334790    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334803    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.334815    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7860b3b-699d-424b-8a3e-ff32cf1a6cda-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.335956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.335385    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-socket" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.335956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.335577    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-data" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.335956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.335761    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-certs" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.336161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.335963    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-credential-socket" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.336644 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.336585    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-socket" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.336748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.336679    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-data" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.337049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.336855    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-credential-socket" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.338023 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.337417    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istiod-ca-cert" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:33.338112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.338069    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-certs" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.338354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.338330    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab95c54-be84-4760-ab71-1d017403a629-istiod-ca-cert" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:33.341029 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.340999    4429 generic.go:356] "Generic (PLEG): container finished" podID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerID="2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df" exitCode=0
Oct 30 13:17:33.341134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.341099    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" event={"ID":"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5","Type":"ContainerDied","Data":"2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df"}
Oct 30 13:17:33.341206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.341147    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c" event={"ID":"c7cf78f7-11ac-490e-bbf2-b1eb66f542b5","Type":"ContainerDied","Data":"bf22f35c9d9e3759b458660ee22a2dc1381c7db99e9e53065766de7b3894b3b8"}
Oct 30 13:17:33.342816 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.341369    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"
Oct 30 13:17:33.346936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.346909    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-envoy" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.347913 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.347488    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11535-cc7e99" event={"ID":"a7531b42-0d02-41db-92d7-feff6d48cec1","Type":"ContainerStarted","Data":"dcbc701cff406d413a661e65ea8adeb7ace3bf2b58a043050229aa5855b3c8a5"}
Oct 30 13:17:33.347913 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.347747    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-kube-api-access-mjt98" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "kube-api-access-mjt98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:33.348307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.348264    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-kube-api-access-t7v89" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "kube-api-access-t7v89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:33.348409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.348269    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-envoy" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:33.349465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.349427    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-podinfo" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:33.349907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.349830    4429 scope.go:122] "RemoveContainer" containerID="c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3"
Oct 30 13:17:33.350510 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:33.350382    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3\": container with ID starting with c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3 not found: ID does not exist" containerID="c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3"
Oct 30 13:17:33.350510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.350417    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3"} err="failed to get container status \"c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3\": rpc error: code = NotFound desc = could not find container \"c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3\": container with ID starting with c545d4a6240563ce9a76bb5d32cbd749ed65ebb926430908c20c1adb353325c3 not found: ID does not exist"
Oct 30 13:17:33.350510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.350438    4429 scope.go:122] "RemoveContainer" containerID="79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f"
Oct 30 13:17:33.351560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.351422    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-istio-token" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:33.351724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.351702    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cab95c54-be84-4760-ab71-1d017403a629-istio-podinfo" pod "cab95c54-be84-4760-ab71-1d017403a629" (UID: "cab95c54-be84-4760-ab71-1d017403a629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:33.356671 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.356645    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-token" pod "f55cd737-74cc-4679-8dd5-fa41d82dbc47" (UID: "f55cd737-74cc-4679-8dd5-fa41d82dbc47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:33.379069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.379042    4429 scope.go:122] "RemoveContainer" containerID="79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f"
Oct 30 13:17:33.379883 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:33.379617    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f\": container with ID starting with 79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f not found: ID does not exist" containerID="79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f"
Oct 30 13:17:33.379883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.379660    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f"} err="failed to get container status \"79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f\": rpc error: code = NotFound desc = could not find container \"79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f\": container with ID starting with 79a38559b58363d0415ccb68f1860eadfb534ffdba726ce48c6fa9972d8d4b0f not found: ID does not exist"
Oct 30 13:17:33.379883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.379685    4429 scope.go:122] "RemoveContainer" containerID="9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca"
Oct 30 13:17:33.391567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.391487    4429 scope.go:122] "RemoveContainer" containerID="9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca"
Oct 30 13:17:33.391903 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:33.391877    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca\": container with ID starting with 9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca not found: ID does not exist" containerID="9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca"
Oct 30 13:17:33.391996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.391916    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca"} err="failed to get container status \"9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca\": rpc error: code = NotFound desc = could not find container \"9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca\": container with ID starting with 9df92acd969391d2a189fb58f6f1950cf227d22206ec5e6c2878e24eae8a21ca not found: ID does not exist"
Oct 30 13:17:33.391996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.391939    4429 scope.go:122] "RemoveContainer" containerID="2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df"
Oct 30 13:17:33.403342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.403317    4429 scope.go:122] "RemoveContainer" containerID="2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df"
Oct 30 13:17:33.403736 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:33.403708    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df\": container with ID starting with 2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df not found: ID does not exist" containerID="2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df"
Oct 30 13:17:33.403859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.403741    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df"} err="failed to get container status \"2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df\": rpc error: code = NotFound desc = could not find container \"2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df\": container with ID starting with 2d9fc580397022a925ddfd5afc1b48d944c1cba2f4e891ba5589bc083cfe21df not found: ID does not exist"
Oct 30 13:17:33.436357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436324    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436354    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436364    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436373    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjt98\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-kube-api-access-mjt98\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436382    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cab95c54-be84-4760-ab71-1d017403a629-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436390    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436397    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436406    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436414    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cab95c54-be84-4760-ab71-1d017403a629-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436422    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436430    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436437    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436445    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cab95c54-be84-4760-ab71-1d017403a629-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436454    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7v89\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-kube-api-access-t7v89\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436462    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55cd737-74cc-4679-8dd5-fa41d82dbc47-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436469    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55cd737-74cc-4679-8dd5-fa41d82dbc47-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436477    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.436620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.436485    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cab95c54-be84-4760-ab71-1d017403a629-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:33.473149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.473082    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"]
Oct 30 13:17:33.481119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.481089    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-f7bb2"]
Oct 30 13:17:33.519964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.519931    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"]
Oct 30 13:17:33.523202 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.523179    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-jn47c"]
Oct 30 13:17:33.589112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.589077    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:17:33.589273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.589143    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:17:33.673984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.673890    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"]
Oct 30 13:17:33.681946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.681916    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lk7fk"]
Oct 30 13:17:33.717608 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.717574    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"]
Oct 30 13:17:33.721694 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:33.721670    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7a8b03-openshift-default-85fbd85cdc-lb4nk"]
Oct 30 13:17:34.076040 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.075999    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" path="/var/lib/kubelet/pods/c7cf78f7-11ac-490e-bbf2-b1eb66f542b5/volumes"
Oct 30 13:17:34.076579 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.076557    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab95c54-be84-4760-ab71-1d017403a629" path="/var/lib/kubelet/pods/cab95c54-be84-4760-ab71-1d017403a629/volumes"
Oct 30 13:17:34.077054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.077035    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" path="/var/lib/kubelet/pods/f55cd737-74cc-4679-8dd5-fa41d82dbc47/volumes"
Oct 30 13:17:34.077869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.077851    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" path="/var/lib/kubelet/pods/f7860b3b-699d-424b-8a3e-ff32cf1a6cda/volumes"
Oct 30 13:17:34.286371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.286334    4429 patch_prober.go:29] interesting pod/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:17:34.286775 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.286402    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:17:34.358606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.358513    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11535-cc7e99" event={"ID":"a7531b42-0d02-41db-92d7-feff6d48cec1","Type":"ContainerStarted","Data":"45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4"}
Oct 30 13:17:34.404527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.404479    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11535-cc7e99" podStartSLOduration=2.404465123 podStartE2EDuration="2.404465123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:34.401483445 +0000 UTC m=+2712.456977270" watchObservedRunningTime="2026-10-30 13:17:34.404465123 +0000 UTC m=+2712.459958946"
Oct 30 13:17:34.808019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.807994    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 13:17:34.853128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.853099    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:17:34.951560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.951532    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 13:17:34.955905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.955878    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 13:17:34.966794 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.966770    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-kube-api-access-rmf6s\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-kube-api-access-rmf6s\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.966938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.966801    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-envoy\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.966938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.966826    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-socket\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.966938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.966888    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-istio-token\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-istio-token\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.966938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.966923    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-envoy\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.967169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.966993    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-socket\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.967169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967025    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-certs\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.967169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967107    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-data\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.967169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967142    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d631b85a-dadd-4bfa-8460-7955363b5824-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d631b85a-dadd-4bfa-8460-7955363b5824-istio-podinfo\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.967405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967171    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-socket" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.967405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967181    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-data\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.967405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967262    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-token\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-token\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.967405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967347    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-credential-socket\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.967405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967370    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-data" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.967658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967432    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-podinfo\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.967658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967469    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-credential-socket\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.967658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967527    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-socket" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.967658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967536    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-data" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967882    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-certs" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.967620    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-kube-api-access-jkvh9\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-kube-api-access-jkvh9\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.968378    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ef9e63-2828-4881-89e8-a00b74c5c68a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ef9e63-2828-4881-89e8-a00b74c5c68a-istiod-ca-cert\") pod \"11ef9e63-2828-4881-89e8-a00b74c5c68a\" (UID: \"11ef9e63-2828-4881-89e8-a00b74c5c68a\") "
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.968455    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d631b85a-dadd-4bfa-8460-7955363b5824-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d631b85a-dadd-4bfa-8460-7955363b5824-istiod-ca-cert\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.968494    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-certs\") pod \"d631b85a-dadd-4bfa-8460-7955363b5824\" (UID: \"d631b85a-dadd-4bfa-8460-7955363b5824\") "
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.969219    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.969240    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.969256    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.969269    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.969475    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-credential-socket" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.969790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.969782    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:34.970627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.969974    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-certs" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.970627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.970176    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ef9e63-2828-4881-89e8-a00b74c5c68a-istiod-ca-cert" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:34.970627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.970417    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d631b85a-dadd-4bfa-8460-7955363b5824-istiod-ca-cert" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:34.970791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.970718    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-envoy" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.971320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.971222    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d631b85a-dadd-4bfa-8460-7955363b5824-istio-podinfo" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:34.971698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.971664    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-envoy" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.971698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.971681    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-credential-socket" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:34.971863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.971849    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-kube-api-access-jkvh9" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "kube-api-access-jkvh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:34.971930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.971873    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-token" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:34.972005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.971985    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-kube-api-access-rmf6s" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "kube-api-access-rmf6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:34.972114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.972092    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-istio-token" pod "d631b85a-dadd-4bfa-8460-7955363b5824" (UID: "d631b85a-dadd-4bfa-8460-7955363b5824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:34.972801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:34.972782    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-podinfo" pod "11ef9e63-2828-4881-89e8-a00b74c5c68a" (UID: "11ef9e63-2828-4881-89e8-a00b74c5c68a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.071764    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-certs\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.071827    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-kube-api-access-b788p\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-kube-api-access-b788p\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.071876    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-token\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-token\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.071928    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-data\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.071999    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-envoy\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.073487    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-data" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.073560    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-certs" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.073694    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istiod-ca-cert\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.073759    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-credential-socket\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.073788    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istiod-ca-cert\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.073846    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-kube-api-access-d429j\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-kube-api-access-d429j\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.073893    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-data\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.073968    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-certs\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.074021    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-token\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-token\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.074076    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-credential-socket\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.074139    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-socket\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.074202    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-podinfo\") pod \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\" (UID: \"ea91e557-d41c-4ffa-bd7a-ece09cc765f9\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.074257    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-socket\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.074345    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-podinfo\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.074382    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-envoy\") pod \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\" (UID: \"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd\") "
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.074875    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-certs" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075196    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istiod-ca-cert" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075553    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmf6s\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-kube-api-access-rmf6s\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075578    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075604    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075619    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d631b85a-dadd-4bfa-8460-7955363b5824-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075635    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075652    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075672    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d631b85a-dadd-4bfa-8460-7955363b5824-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075686    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075701    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ef9e63-2828-4881-89e8-a00b74c5c68a-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075716    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ef9e63-2828-4881-89e8-a00b74c5c68a-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075735    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075751    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvh9\" (UniqueName: \"kubernetes.io/projected/11ef9e63-2828-4881-89e8-a00b74c5c68a-kube-api-access-jkvh9\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075767    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075788    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ef9e63-2828-4881-89e8-a00b74c5c68a-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075801    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075818    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d631b85a-dadd-4bfa-8460-7955363b5824-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.075832    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d631b85a-dadd-4bfa-8460-7955363b5824-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.076180    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-credential-socket" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.077224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.076518    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istiod-ca-cert" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:35.089748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.079308    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-token" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:35.089748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.079394    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-kube-api-access-d429j" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "kube-api-access-d429j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:35.089748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.079560    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-credential-socket" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.089748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.079586    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-data" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.089748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.079776    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-socket" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.089748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.079775    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-socket" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.089748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.082233    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-podinfo" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:35.097881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.092872    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-envoy" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.097881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.093304    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-envoy" pod "ea91e557-d41c-4ffa-bd7a-ece09cc765f9" (UID: "ea91e557-d41c-4ffa-bd7a-ece09cc765f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:35.097881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.093335    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-token" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:35.099757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.099716    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-kube-api-access-b788p" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "kube-api-access-b788p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:35.101699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.101653    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-podinfo" pod "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" (UID: "4d5795dc-ef5d-4eaf-8505-b2ad430adfcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:35.177098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177015    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177050    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177065    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177079    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177093    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177106    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177119    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b788p\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-kube-api-access-b788p\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177134    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177147    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177158    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177172    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177186    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d429j\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-kube-api-access-d429j\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177198    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.177523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.177212    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea91e557-d41c-4ffa-bd7a-ece09cc765f9-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.370257 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.370212    4429 generic.go:356] "Generic (PLEG): container finished" podID="d631b85a-dadd-4bfa-8460-7955363b5824" containerID="fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae" exitCode=0
Oct 30 13:17:35.370754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.370255    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" event={"ID":"d631b85a-dadd-4bfa-8460-7955363b5824","Type":"ContainerDied","Data":"fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae"}
Oct 30 13:17:35.370754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.370297    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"
Oct 30 13:17:35.370754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.370366    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp" event={"ID":"d631b85a-dadd-4bfa-8460-7955363b5824","Type":"ContainerDied","Data":"4c39cff382b295e97ec4a96898a62a65eb4e804c903ae7aa1eb515eef0d21e2f"}
Oct 30 13:17:35.370754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.370381    4429 scope.go:122] "RemoveContainer" containerID="fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae"
Oct 30 13:17:35.372237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.372210    4429 generic.go:356] "Generic (PLEG): container finished" podID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerID="dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336" exitCode=0
Oct 30 13:17:35.372385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.372269    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" event={"ID":"11ef9e63-2828-4881-89e8-a00b74c5c68a","Type":"ContainerDied","Data":"dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336"}
Oct 30 13:17:35.372385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.372324    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"
Oct 30 13:17:35.372385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.372338    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr" event={"ID":"11ef9e63-2828-4881-89e8-a00b74c5c68a","Type":"ContainerDied","Data":"5a7476bbabbbc4fbacfdec64323e882487a5a6b9e9bae41fdc63ae1f93040347"}
Oct 30 13:17:35.374136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.374112    4429 generic.go:356] "Generic (PLEG): container finished" podID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerID="f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92" exitCode=0
Oct 30 13:17:35.374235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.374187    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"
Oct 30 13:17:35.374235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.374188    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" event={"ID":"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd","Type":"ContainerDied","Data":"f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92"}
Oct 30 13:17:35.374391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.374261    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g" event={"ID":"4d5795dc-ef5d-4eaf-8505-b2ad430adfcd","Type":"ContainerDied","Data":"6be9963f5df29ccab72235a0955e240e3e34c736003ece452a9ec7e84de5d14d"}
Oct 30 13:17:35.375826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.375803    4429 generic.go:356] "Generic (PLEG): container finished" podID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerID="d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a" exitCode=0
Oct 30 13:17:35.375922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.375839    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" event={"ID":"ea91e557-d41c-4ffa-bd7a-ece09cc765f9","Type":"ContainerDied","Data":"d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a"}
Oct 30 13:17:35.375922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.375878    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"
Oct 30 13:17:35.375922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.375883    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm" event={"ID":"ea91e557-d41c-4ffa-bd7a-ece09cc765f9","Type":"ContainerDied","Data":"4a32e9871501142ce9b9a9e7c8014928e052bd79d4eebe84eecb837cf2bf906a"}
Oct 30 13:17:35.384199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.384183    4429 scope.go:122] "RemoveContainer" containerID="fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae"
Oct 30 13:17:35.384547 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:35.384521    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae\": container with ID starting with fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae not found: ID does not exist" containerID="fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae"
Oct 30 13:17:35.384640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.384558    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae"} err="failed to get container status \"fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae\": rpc error: code = NotFound desc = could not find container \"fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae\": container with ID starting with fe59c9617b0ff0221e09e52b38c948f0c6f8501b355ef25fdb245e4d887f62ae not found: ID does not exist"
Oct 30 13:17:35.384640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.384582    4429 scope.go:122] "RemoveContainer" containerID="dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336"
Oct 30 13:17:35.396629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.396267    4429 scope.go:122] "RemoveContainer" containerID="dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336"
Oct 30 13:17:35.398525 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:35.398506    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336\": container with ID starting with dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336 not found: ID does not exist" containerID="dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336"
Oct 30 13:17:35.398611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.398530    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336"} err="failed to get container status \"dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336\": rpc error: code = NotFound desc = could not find container \"dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336\": container with ID starting with dd199bb3c2d66bd0c309b897d8c6bb698ef30215d59cf514d6c94468092a8336 not found: ID does not exist"
Oct 30 13:17:35.398611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.398545    4429 scope.go:122] "RemoveContainer" containerID="f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92"
Oct 30 13:17:35.408162 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.408144    4429 scope.go:122] "RemoveContainer" containerID="f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92"
Oct 30 13:17:35.408523 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:35.408505    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92\": container with ID starting with f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92 not found: ID does not exist" containerID="f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92"
Oct 30 13:17:35.408611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.408528    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92"} err="failed to get container status \"f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92\": rpc error: code = NotFound desc = could not find container \"f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92\": container with ID starting with f7eff03b5864080928be58357e592eabfcfd0f8f1c50228527bfe68227cb4a92 not found: ID does not exist"
Oct 30 13:17:35.408611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.408543    4429 scope.go:122] "RemoveContainer" containerID="d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a"
Oct 30 13:17:35.409100 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.409085    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11535-cc7e99"]
Oct 30 13:17:35.417541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.417525    4429 scope.go:122] "RemoveContainer" containerID="d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a"
Oct 30 13:17:35.417834 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:35.417811    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a\": container with ID starting with d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a not found: ID does not exist" containerID="d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a"
Oct 30 13:17:35.417922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.417836    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a"} err="failed to get container status \"d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a\": rpc error: code = NotFound desc = could not find container \"d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a\": container with ID starting with d3a3509ed166a2e0cd1ff6a75906f4c416330bdea3c10b4f2818f08ae1d68a7a not found: ID does not exist"
Oct 30 13:17:35.434917 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.434892    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11535-cc7e99"]
Oct 30 13:17:35.444413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.444386    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"]
Oct 30 13:17:35.477336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.477308    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-wwm5g"]
Oct 30 13:17:35.516520 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.516483    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"]
Oct 30 13:17:35.520613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.520584    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-c5jnr"]
Oct 30 13:17:35.558480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.558445    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"]
Oct 30 13:17:35.567603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.567579    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-gt7rp"]
Oct 30 13:17:35.595333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.595273    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11535-e19d44"]
Oct 30 13:17:35.595978 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.595960    4429 cpu_manager.go:414] "containerMap: removing container" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.595979    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.595998    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a7531b42-0d02-41db-92d7-feff6d48cec1" containerName="test-curl-11535-cc7e99"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596003    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7531b42-0d02-41db-92d7-feff6d48cec1" containerName="test-curl-11535-cc7e99"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596014    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596019    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596027    4429 cpu_manager.go:414] "containerMap: removing container" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596032    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596038    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596043    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596069    4429 cpu_manager.go:414] "containerMap: removing container" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy"
Oct 30 13:17:35.596070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596073    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596080    4429 cpu_manager.go:414] "containerMap: removing container" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596084    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596093    4429 cpu_manager.go:414] "containerMap: removing container" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596101    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596111    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596115    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596131    4429 cpu_manager.go:414] "containerMap: removing container" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596139    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596228    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55cd737-74cc-4679-8dd5-fa41d82dbc47" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596239    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db73b6-604c-4d87-b93a-e03a2f68e0ee" containerName="cleanup"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596247    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7531b42-0d02-41db-92d7-feff6d48cec1" containerName="test-curl-11535-cc7e99"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596254    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7cf78f7-11ac-490e-bbf2-b1eb66f542b5" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596261    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596266    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596274    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab95c54-be84-4760-ab71-1d017403a629" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596279    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596300    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" containerName="istio-proxy"
Oct 30 13:17:35.596545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.596307    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7860b3b-699d-424b-8a3e-ff32cf1a6cda" containerName="istio-proxy"
Oct 30 13:17:35.601558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.601535    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:35.605554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.605527    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"]
Oct 30 13:17:35.608861 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.608843    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-bb86fd-openshift-default-67498875d9-57bjm"]
Oct 30 13:17:35.617588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.617563    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11535-e19d44"]
Oct 30 13:17:35.620817 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.620794    4429 patch_prober.go:29] interesting pod/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:17:35.620916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.620835    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:17:35.684031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.683998    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jspx\" (UniqueName: \"kubernetes.io/projected/ab719fc0-7280-4974-9454-3936809dc2df-kube-api-access-4jspx\") pod \"test-curl-11535-e19d44\" (UID: \"ab719fc0-7280-4974-9454-3936809dc2df\") " pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:35.785868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.785785    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jspx\" (UniqueName: \"kubernetes.io/projected/ab719fc0-7280-4974-9454-3936809dc2df-kube-api-access-4jspx\") pod \"test-curl-11535-e19d44\" (UID: \"ab719fc0-7280-4974-9454-3936809dc2df\") " pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:35.811482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.811455    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jspx\" (UniqueName: \"kubernetes.io/projected/ab719fc0-7280-4974-9454-3936809dc2df-kube-api-access-4jspx\") pod \"test-curl-11535-e19d44\" (UID: \"ab719fc0-7280-4974-9454-3936809dc2df\") " pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:35.918142 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:35.918108    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:36.075184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.075108    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ef9e63-2828-4881-89e8-a00b74c5c68a" path="/var/lib/kubelet/pods/11ef9e63-2828-4881-89e8-a00b74c5c68a/volumes"
Oct 30 13:17:36.075648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.075632    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5795dc-ef5d-4eaf-8505-b2ad430adfcd" path="/var/lib/kubelet/pods/4d5795dc-ef5d-4eaf-8505-b2ad430adfcd/volumes"
Oct 30 13:17:36.076057 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.076044    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d631b85a-dadd-4bfa-8460-7955363b5824" path="/var/lib/kubelet/pods/d631b85a-dadd-4bfa-8460-7955363b5824/volumes"
Oct 30 13:17:36.076802 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.076786    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea91e557-d41c-4ffa-bd7a-ece09cc765f9" path="/var/lib/kubelet/pods/ea91e557-d41c-4ffa-bd7a-ece09cc765f9/volumes"
Oct 30 13:17:36.225216 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:17:36.225182    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab719fc0_7280_4974_9454_3936809dc2df.slice/crio-c0d72bb40d4e73ff70e92b6d88878da6ef334fee3ac2932e07436ba25210d35b WatchSource:0}: Error finding container c0d72bb40d4e73ff70e92b6d88878da6ef334fee3ac2932e07436ba25210d35b: Status 404 returned error can't find the container with id c0d72bb40d4e73ff70e92b6d88878da6ef334fee3ac2932e07436ba25210d35b
Oct 30 13:17:36.225639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.225620    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11535-e19d44"]
Oct 30 13:17:36.388433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.388356    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11535-e19d44" event={"ID":"ab719fc0-7280-4974-9454-3936809dc2df","Type":"ContainerStarted","Data":"e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195"}
Oct 30 13:17:36.388433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.388396    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11535-e19d44" event={"ID":"ab719fc0-7280-4974-9454-3936809dc2df","Type":"ContainerStarted","Data":"c0d72bb40d4e73ff70e92b6d88878da6ef334fee3ac2932e07436ba25210d35b"}
Oct 30 13:17:36.389663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.389638    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11535-cc7e99" podUID="a7531b42-0d02-41db-92d7-feff6d48cec1" containerName="test-curl-11535-cc7e99" containerID="cri-o://45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4" gracePeriod=2
Oct 30 13:17:36.429992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.429943    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11535-e19d44" podStartSLOduration=1.4299299730000001 podStartE2EDuration="1.429929973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:36.42972462 +0000 UTC m=+2714.485218445" watchObservedRunningTime="2026-10-30 13:17:36.429929973 +0000 UTC m=+2714.485423798"
Oct 30 13:17:36.748819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.748783    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:17:36.908756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.908719    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istiod-ca-cert\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.908918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.908779    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-socket\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.908918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.908800    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-envoy\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.908918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.908869    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-credential-socket\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.909087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.908918    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-podinfo\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.909087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.908955    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-token\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-token\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.909087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909004    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-data\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.909087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909040    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-kube-api-access-t7khh\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-kube-api-access-t7khh\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.909313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909137    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-socket" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:36.909313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909238    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-credential-socket" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:36.909313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909157    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istiod-ca-cert" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:36.909313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909271    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-certs\") pod \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\" (UID: \"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7\") "
Oct 30 13:17:36.909536 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909476    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-certs" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:36.909594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909576    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-data" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:36.909879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909859    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:36.909960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909879    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:36.909960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909889    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:36.909960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909898    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:36.909960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.909906    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:36.911507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.911468    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-token" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:36.911784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.911507    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-podinfo" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:36.911985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.911961    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-envoy" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:36.912188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:36.912170    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-kube-api-access-t7khh" pod "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" (UID: "b0bea9c3-b4f0-44be-a392-fa04d29a6dc7"). InnerVolumeSpecName "kube-api-access-t7khh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:37.011361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.011332    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:37.011361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.011357    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:37.011542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.011370    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:37.011542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.011383    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7khh\" (UniqueName: \"kubernetes.io/projected/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7-kube-api-access-t7khh\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:37.391699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.391607    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11535-e19d44"]
Oct 30 13:17:37.395532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.395505    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11535-e19d44"]
Oct 30 13:17:37.402442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.402406    4429 generic.go:356] "Generic (PLEG): container finished" podID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerID="1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f" exitCode=0
Oct 30 13:17:37.403447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.403425    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"
Oct 30 13:17:37.407387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.407359    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" event={"ID":"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7","Type":"ContainerDied","Data":"1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f"}
Oct 30 13:17:37.407498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.407420    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km" event={"ID":"b0bea9c3-b4f0-44be-a392-fa04d29a6dc7","Type":"ContainerDied","Data":"73058e543e60f2e1312e619253b69b0a3f2e244d4dcbb489360343fa0b1f5805"}
Oct 30 13:17:37.407498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.407451    4429 scope.go:122] "RemoveContainer" containerID="1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f"
Oct 30 13:17:37.423026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.423003    4429 scope.go:122] "RemoveContainer" containerID="461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c"
Oct 30 13:17:37.433442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.433420    4429 scope.go:122] "RemoveContainer" containerID="1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f"
Oct 30 13:17:37.433745 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:37.433710    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f\": container with ID starting with 1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f not found: ID does not exist" containerID="1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f"
Oct 30 13:17:37.433846 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.433758    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f"} err="failed to get container status \"1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f\": rpc error: code = NotFound desc = could not find container \"1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f\": container with ID starting with 1ccbdf436e901905d45024bc4f33115df54717cd9cded080a670f7911cec8a3f not found: ID does not exist"
Oct 30 13:17:37.433846 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.433784    4429 scope.go:122] "RemoveContainer" containerID="461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c"
Oct 30 13:17:37.434073 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:37.434047    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c\": container with ID starting with 461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c not found: ID does not exist" containerID="461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c"
Oct 30 13:17:37.434120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.434082    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c"} err="failed to get container status \"461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c\": rpc error: code = NotFound desc = could not find container \"461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c\": container with ID starting with 461dfd18876aa7a00b84cea1cd9ad04971b126f4cded670a41fbf476914efc0c not found: ID does not exist"
Oct 30 13:17:37.465211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.465174    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"]
Oct 30 13:17:37.479373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:37.479347    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ca097a-openshift-default-7478b86885-578km"]
Oct 30 13:17:38.076742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.076712    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" path="/var/lib/kubelet/pods/b0bea9c3-b4f0-44be-a392-fa04d29a6dc7/volumes"
Oct 30 13:17:38.410973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.410930    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11535-e19d44" podUID="ab719fc0-7280-4974-9454-3936809dc2df" containerName="test-curl-11535-e19d44" containerID="cri-o://e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195" gracePeriod=2
Oct 30 13:17:38.510928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.510868    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"]
Oct 30 13:17:38.511795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.511770    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ab719fc0-7280-4974-9454-3936809dc2df" containerName="test-curl-11535-e19d44"
Oct 30 13:17:38.511795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.511790    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab719fc0-7280-4974-9454-3936809dc2df" containerName="test-curl-11535-e19d44"
Oct 30 13:17:38.512015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.511827    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy"
Oct 30 13:17:38.512015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.511836    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy"
Oct 30 13:17:38.512015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.511865    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy"
Oct 30 13:17:38.512015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.511872    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy"
Oct 30 13:17:38.512015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.511995    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy"
Oct 30 13:17:38.512015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.512009    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bea9c3-b4f0-44be-a392-fa04d29a6dc7" containerName="istio-proxy"
Oct 30 13:17:38.512354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.512023    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab719fc0-7280-4974-9454-3936809dc2df" containerName="test-curl-11535-e19d44"
Oct 30 13:17:38.516269 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.516247    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.542507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.542477    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"]
Oct 30 13:17:38.551793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.551764    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f49b40-openshift-default-dockercfg-8kzdx\""
Oct 30 13:17:38.623772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.623748    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11535-cc7e99"
Oct 30 13:17:38.631914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.631892    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-istio-token\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.632039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.631951    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-socket\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.632087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.632059    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6dd987-feca-427c-8f9b-724c899c2678-istio-podinfo\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.632138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.632085    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-envoy\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.632227 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.632204    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-credential-socket\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.632325 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.632303    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-data\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.632429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.632399    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-certs\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.632429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.632421    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgkq\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-kube-api-access-ghgkq\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.632513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.632454    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6dd987-feca-427c-8f9b-724c899c2678-istiod-ca-cert\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.735181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.735143    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7531b42-0d02-41db-92d7-feff6d48cec1-kube-api-access-qb68m\" (UniqueName: \"kubernetes.io/projected/a7531b42-0d02-41db-92d7-feff6d48cec1-kube-api-access-qb68m\") pod \"a7531b42-0d02-41db-92d7-feff6d48cec1\" (UID: \"a7531b42-0d02-41db-92d7-feff6d48cec1\") "
Oct 30 13:17:38.735838 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.735807    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-certs\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.735973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.735859    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgkq\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-kube-api-access-ghgkq\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.735973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.735920    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6dd987-feca-427c-8f9b-724c899c2678-istiod-ca-cert\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.735973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.735964    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-istio-token\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.736083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.736034    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-socket\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.736156 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.736138    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6dd987-feca-427c-8f9b-724c899c2678-istio-podinfo\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.736566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.736357    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-envoy\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.736648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.736625    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-credential-socket\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.736721 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.736705    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-data\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.737257 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.737237    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-data\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.738157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.738130    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-credential-socket\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.739149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.738559    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-certs\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.739149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.738774    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6dd987-feca-427c-8f9b-724c899c2678-istiod-ca-cert\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.739149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.739100    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-socket\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.742593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.742552    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7531b42-0d02-41db-92d7-feff6d48cec1-kube-api-access-qb68m" pod "a7531b42-0d02-41db-92d7-feff6d48cec1" (UID: "a7531b42-0d02-41db-92d7-feff6d48cec1"). InnerVolumeSpecName "kube-api-access-qb68m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:38.742933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.742911    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-envoy\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.753339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.744414    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6dd987-feca-427c-8f9b-724c899c2678-istio-podinfo\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.765384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.765354    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-istio-token\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.765507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.765392    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgkq\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-kube-api-access-ghgkq\") pod \"e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") " pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.836965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.836930    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:38.837722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:38.837702    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb68m\" (UniqueName: \"kubernetes.io/projected/a7531b42-0d02-41db-92d7-feff6d48cec1-kube-api-access-qb68m\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:39.159600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.159569    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"]
Oct 30 13:17:39.159867 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:17:39.159843    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6dd987_feca_427c_8f9b_724c899c2678.slice/crio-19e2e95c45fc43619d88e4d02147d412bb1ccd4bac496bb15567fe31d8be2b77 WatchSource:0}: Error finding container 19e2e95c45fc43619d88e4d02147d412bb1ccd4bac496bb15567fe31d8be2b77: Status 404 returned error can't find the container with id 19e2e95c45fc43619d88e4d02147d412bb1ccd4bac496bb15567fe31d8be2b77
Oct 30 13:17:39.162524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.162492    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:39.162596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.162559    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:39.162596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.162587    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:39.418992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.418908    4429 generic.go:356] "Generic (PLEG): container finished" podID="a7531b42-0d02-41db-92d7-feff6d48cec1" containerID="45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4" exitCode=137
Oct 30 13:17:39.418992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.418957    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11535-cc7e99"
Oct 30 13:17:39.419509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.419016    4429 scope.go:122] "RemoveContainer" containerID="45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4"
Oct 30 13:17:39.421059 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.421023    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" event={"ID":"0f6dd987-feca-427c-8f9b-724c899c2678","Type":"ContainerStarted","Data":"7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb"}
Oct 30 13:17:39.421186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.421073    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" event={"ID":"0f6dd987-feca-427c-8f9b-724c899c2678","Type":"ContainerStarted","Data":"19e2e95c45fc43619d88e4d02147d412bb1ccd4bac496bb15567fe31d8be2b77"}
Oct 30 13:17:39.430981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.430966    4429 scope.go:122] "RemoveContainer" containerID="45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4"
Oct 30 13:17:39.431223 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:39.431202    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4\": container with ID starting with 45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4 not found: ID does not exist" containerID="45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4"
Oct 30 13:17:39.431311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.431232    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4"} err="failed to get container status \"45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4\": rpc error: code = NotFound desc = could not find container \"45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4\": container with ID starting with 45e68c493acfce7ca6b330ed7ae3e3845307506bbf3d36bd6777e1859d9c7dc4 not found: ID does not exist"
Oct 30 13:17:39.475796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.475747    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" podStartSLOduration=1.475732659 podStartE2EDuration="1.475732659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:39.474558364 +0000 UTC m=+2717.530052189" watchObservedRunningTime="2026-10-30 13:17:39.475732659 +0000 UTC m=+2717.531226482"
Oct 30 13:17:39.489802 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:39.489773    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11535-cc7e99\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="a7531b42-0d02-41db-92d7-feff6d48cec1" pod="opendatahub/test-curl-11535-cc7e99"
Oct 30 13:17:39.837185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:39.837156    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:40.076033 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:40.075998    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7531b42-0d02-41db-92d7-feff6d48cec1" path="/var/lib/kubelet/pods/a7531b42-0d02-41db-92d7-feff6d48cec1/volumes"
Oct 30 13:17:40.624744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:40.624718    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:40.657811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:40.657777    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab719fc0-7280-4974-9454-3936809dc2df-kube-api-access-4jspx\" (UniqueName: \"kubernetes.io/projected/ab719fc0-7280-4974-9454-3936809dc2df-kube-api-access-4jspx\") pod \"ab719fc0-7280-4974-9454-3936809dc2df\" (UID: \"ab719fc0-7280-4974-9454-3936809dc2df\") "
Oct 30 13:17:40.660041 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:40.660012    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab719fc0-7280-4974-9454-3936809dc2df-kube-api-access-4jspx" pod "ab719fc0-7280-4974-9454-3936809dc2df" (UID: "ab719fc0-7280-4974-9454-3936809dc2df"). InnerVolumeSpecName "kube-api-access-4jspx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:40.759037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:40.759006    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jspx\" (UniqueName: \"kubernetes.io/projected/ab719fc0-7280-4974-9454-3936809dc2df-kube-api-access-4jspx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.843643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:40.843617    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:41.436678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:41.436644    4429 generic.go:356] "Generic (PLEG): container finished" podID="ab719fc0-7280-4974-9454-3936809dc2df" containerID="e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195" exitCode=137
Oct 30 13:17:41.436850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:41.436702    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:41.436850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:41.436742    4429 scope.go:122] "RemoveContainer" containerID="e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195"
Oct 30 13:17:41.437161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:41.437141    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:41.438276 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:41.438261    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:17:41.447533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:41.447514    4429 scope.go:122] "RemoveContainer" containerID="e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195"
Oct 30 13:17:41.447868 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:41.447845    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195\": container with ID starting with e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195 not found: ID does not exist" containerID="e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195"
Oct 30 13:17:41.447923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:41.447882    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195"} err="failed to get container status \"e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195\": rpc error: code = NotFound desc = could not find container \"e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195\": container with ID starting with e5896e08038d5e318b5e39159894d89340d364b4341cae03264eeb1d4c77f195 not found: ID does not exist"
Oct 30 13:17:41.503170 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:41.503125    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11535-e19d44\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="ab719fc0-7280-4974-9454-3936809dc2df" pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:42.075837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:42.075804    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab719fc0-7280-4974-9454-3936809dc2df" path="/var/lib/kubelet/pods/ab719fc0-7280-4974-9454-3936809dc2df/volumes"
Oct 30 13:17:42.085951 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:42.085920    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11535-e19d44\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="ab719fc0-7280-4974-9454-3936809dc2df" pod="opendatahub/test-curl-11535-e19d44"
Oct 30 13:17:42.372240 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:17:42.372159    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:17:43.954749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:43.954719    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"]
Oct 30 13:17:43.960199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:43.960180    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:43.979772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:43.979737    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"]
Oct 30 13:17:43.990560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:43.990534    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f49b40-dockercfg-d42zk\""
Oct 30 13:17:43.993420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:43.993397    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f49b40\""
Oct 30 13:17:44.002078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.002048    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw\" (UID: \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:44.002323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.002268    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pj8l\" (UniqueName: \"kubernetes.io/projected/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-kube-api-access-5pj8l\") pod \"payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw\" (UID: \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:44.104985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.104947    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pj8l\" (UniqueName: \"kubernetes.io/projected/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-kube-api-access-5pj8l\") pod \"payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw\" (UID: \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:44.105172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.105117    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw\" (UID: \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:44.106185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.106100    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw\" (UID: \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:44.111398 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.108726    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"]
Oct 30 13:17:44.130614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.130577    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"]
Oct 30 13:17:44.130794 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.130750    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:44.137985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.137959    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pj8l\" (UniqueName: \"kubernetes.io/projected/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-kube-api-access-5pj8l\") pod \"payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw\" (UID: \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:44.206613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.206541    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2cs\" (UniqueName: \"kubernetes.io/projected/b7562f85-16f9-466c-8814-42b7c8307db1-kube-api-access-lf2cs\") pod \"payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x\" (UID: \"b7562f85-16f9-466c-8814-42b7c8307db1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:44.206613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.206594    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7562f85-16f9-466c-8814-42b7c8307db1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x\" (UID: \"b7562f85-16f9-466c-8814-42b7c8307db1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:44.273905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.273877    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:44.308457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.308427    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2cs\" (UniqueName: \"kubernetes.io/projected/b7562f85-16f9-466c-8814-42b7c8307db1-kube-api-access-lf2cs\") pod \"payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x\" (UID: \"b7562f85-16f9-466c-8814-42b7c8307db1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:44.308613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.308499    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7562f85-16f9-466c-8814-42b7c8307db1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x\" (UID: \"b7562f85-16f9-466c-8814-42b7c8307db1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:44.309205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.309186    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7562f85-16f9-466c-8814-42b7c8307db1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x\" (UID: \"b7562f85-16f9-466c-8814-42b7c8307db1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:44.333888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.333864    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2cs\" (UniqueName: \"kubernetes.io/projected/b7562f85-16f9-466c-8814-42b7c8307db1-kube-api-access-lf2cs\") pod \"payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x\" (UID: \"b7562f85-16f9-466c-8814-42b7c8307db1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:44.469451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.469376    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:44.584058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.584026    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"]
Oct 30 13:17:44.782423 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:17:44.782392    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7562f85_16f9_466c_8814_42b7c8307db1.slice/crio-74ae2d5b2be07cdca1ebe5e67a628cca4cc23cfcb9af6fce9c2cc9ee1ef67248 WatchSource:0}: Error finding container 74ae2d5b2be07cdca1ebe5e67a628cca4cc23cfcb9af6fce9c2cc9ee1ef67248: Status 404 returned error can't find the container with id 74ae2d5b2be07cdca1ebe5e67a628cca4cc23cfcb9af6fce9c2cc9ee1ef67248
Oct 30 13:17:44.782992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:44.782972    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"]
Oct 30 13:17:45.465096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:45.465056    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw" event={"ID":"bc86c779-342a-4ad1-bbe5-8a3cda3983a3","Type":"ContainerStarted","Data":"358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93"}
Oct 30 13:17:45.465521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:45.465129    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw" event={"ID":"bc86c779-342a-4ad1-bbe5-8a3cda3983a3","Type":"ContainerStarted","Data":"c1e82f3577e801d3d6cccca3cc875a8c492e3a9a6068e50986bfb703d4255310"}
Oct 30 13:17:45.465521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:45.465300    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:45.467018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:45.466994    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x" event={"ID":"b7562f85-16f9-466c-8814-42b7c8307db1","Type":"ContainerStarted","Data":"187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0"}
Oct 30 13:17:45.467122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:45.467033    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x" event={"ID":"b7562f85-16f9-466c-8814-42b7c8307db1","Type":"ContainerStarted","Data":"74ae2d5b2be07cdca1ebe5e67a628cca4cc23cfcb9af6fce9c2cc9ee1ef67248"}
Oct 30 13:17:45.467122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:45.467053    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:17:45.511916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:45.511873    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw" podStartSLOduration=2.511859564 podStartE2EDuration="2.511859564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:45.510115854 +0000 UTC m=+2723.565609679" watchObservedRunningTime="2026-10-30 13:17:45.511859564 +0000 UTC m=+2723.567353420"
Oct 30 13:17:45.555699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:45.555651    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x" podStartSLOduration=1.555636641 podStartE2EDuration="1.555636641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:45.553744514 +0000 UTC m=+2723.609238341" watchObservedRunningTime="2026-10-30 13:17:45.555636641 +0000 UTC m=+2723.611130522"
Oct 30 13:17:56.476452 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:56.476417    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:17:56.476949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:17:56.476500    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:18:02.025560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.025520    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"]
Oct 30 13:18:02.030764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.030735    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056020 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.055989    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nwtx\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-kube-api-access-5nwtx\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056187 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.056167    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-envoy\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.056204    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65162b68-577e-4215-a40d-d6232717b104-istiod-ca-cert\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.056390    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65162b68-577e-4215-a40d-d6232717b104-istio-podinfo\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056565 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.056448    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-data\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.056601    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-credential-socket\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.056667    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-istio-token\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.056720    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-socket\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.056760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.056738    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-certs\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.058527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.058507    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"]
Oct 30 13:18:02.065702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.065675    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-f1af9c-openshift-default-dockercfg-shbbf\""
Oct 30 13:18:02.157844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.157810    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65162b68-577e-4215-a40d-d6232717b104-istio-podinfo\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.157857    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-data\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.157914    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-credential-socket\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.157938    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-istio-token\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.157956    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-socket\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.157972    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-certs\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.158018    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwtx\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-kube-api-access-5nwtx\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.158148    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-envoy\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.158172    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65162b68-577e-4215-a40d-d6232717b104-istiod-ca-cert\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.158431    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-socket\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.158470    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-data\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.158526    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-certs\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.158671    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-credential-socket\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.158921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.158904    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65162b68-577e-4215-a40d-d6232717b104-istiod-ca-cert\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.160615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.160592    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65162b68-577e-4215-a40d-d6232717b104-istio-podinfo\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.160804 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.160788    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-envoy\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.183806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.183783    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-istio-token\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.183806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.183793    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwtx\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-kube-api-access-5nwtx\") pod \"e2e-noten-f1af9c-openshift-default-695c446b69-w72kh\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") " pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.348319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.348211    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:02.675069 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:18:02.675038    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65162b68_577e_4215_a40d_d6232717b104.slice/crio-db690f9fff90db61637225efa673fe0493f6df57c13ce491a4eed6156caeacb7 WatchSource:0}: Error finding container db690f9fff90db61637225efa673fe0493f6df57c13ce491a4eed6156caeacb7: Status 404 returned error can't find the container with id db690f9fff90db61637225efa673fe0493f6df57c13ce491a4eed6156caeacb7
Oct 30 13:18:02.675929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.675901    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"]
Oct 30 13:18:02.677584 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.677552    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:02.677700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.677616    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:02.677700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:02.677644    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:03.590705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:03.590622    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh" event={"ID":"65162b68-577e-4215-a40d-d6232717b104","Type":"ContainerStarted","Data":"2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544"}
Oct 30 13:18:03.590705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:03.590682    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh" event={"ID":"65162b68-577e-4215-a40d-d6232717b104","Type":"ContainerStarted","Data":"db690f9fff90db61637225efa673fe0493f6df57c13ce491a4eed6156caeacb7"}
Oct 30 13:18:03.661130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:03.661083    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh" podStartSLOduration=2.661070744 podStartE2EDuration="2.661070744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:03.65786984 +0000 UTC m=+2741.713363686" watchObservedRunningTime="2026-10-30 13:18:03.661070744 +0000 UTC m=+2741.716564568"
Oct 30 13:18:04.349450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:04.349414    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:04.355157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:04.355135    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:04.598607 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:04.598570    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:04.599688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:04.599638    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:06.593070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.593035    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"]
Oct 30 13:18:06.597639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.597617    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.618482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.618438    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"]
Oct 30 13:18:06.635622 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.635591    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f49b40-serving-cert\""
Oct 30 13:18:06.643642 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.643604    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.643797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.643776    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.643903 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.643882    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.643972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.643920    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntgj8\" (UniqueName: \"kubernetes.io/projected/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-kube-api-access-ntgj8\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.745343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.745264    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.745725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.745391    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.745725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.745434    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgj8\" (UniqueName: \"kubernetes.io/projected/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-kube-api-access-ntgj8\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.745725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.745609    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.746121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.746095    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.746319 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:18:06.746190    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-serving-cert: secret "maas-api-e2e-shared-a-w3-f49b40-serving-cert" not found
Oct 30 13:18:06.746480 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:18:06.746465    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls podName:4f1282ee-48bd-49fa-8af5-5c22dc7abfc6 nodeName:}" failed. No retries permitted until 2026-10-30 13:18:07.246446338 +0000 UTC m=+2745.301940140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls") pod "maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675" (UID: "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6") : secret "maas-api-e2e-shared-a-w3-f49b40-serving-cert" not found
Oct 30 13:18:06.749002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.748979    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:06.771613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:06.771584    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgj8\" (UniqueName: \"kubernetes.io/projected/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-kube-api-access-ntgj8\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:07.254019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:07.253978    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:07.256557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:07.256533    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:07.513753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:07.513658    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:07.831371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:07.831233    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"]
Oct 30 13:18:07.836749 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:18:07.836718    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1282ee_48bd_49fa_8af5_5c22dc7abfc6.slice/crio-5383a5a27bd20f66a2443c96eeaf8f9f1734013e86cc810e7f5be71a64ad7414 WatchSource:0}: Error finding container 5383a5a27bd20f66a2443c96eeaf8f9f1734013e86cc810e7f5be71a64ad7414: Status 404 returned error can't find the container with id 5383a5a27bd20f66a2443c96eeaf8f9f1734013e86cc810e7f5be71a64ad7414
Oct 30 13:18:07.837884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:07.837863    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:18:08.632684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:08.632600    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675" event={"ID":"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6","Type":"ContainerStarted","Data":"b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d"}
Oct 30 13:18:08.632684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:08.632661    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675" event={"ID":"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6","Type":"ContainerStarted","Data":"5383a5a27bd20f66a2443c96eeaf8f9f1734013e86cc810e7f5be71a64ad7414"}
Oct 30 13:18:08.633119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:08.632705    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:08.682146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:08.682097    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675" podStartSLOduration=2.210466911 podStartE2EDuration="2.682083749s" totalImagesPullingTime="471.616838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:08.679734841 +0000 UTC m=+2746.735228670" watchObservedRunningTime="2026-10-30 13:18:08.682083749 +0000 UTC m=+2746.737577597"
Oct 30 13:18:12.377231 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:18:12.377190    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:18:14.645747 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:14.645705    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:18:32.134852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:32.134815    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"]
Oct 30 13:18:32.135343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:32.135078    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh" podUID="65162b68-577e-4215-a40d-d6232717b104" containerName="istio-proxy" containerID="cri-o://2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544" gracePeriod=30
Oct 30 13:18:34.600725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:34.600689    4429 patch_prober.go:29] interesting pod/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:18:34.601209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:34.600765    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh" podUID="65162b68-577e-4215-a40d-d6232717b104" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:18:37.487011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.486981    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:37.603887 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.603852    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-certs\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.603949    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-data\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604037    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-kube-api-access-5nwtx\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-kube-api-access-5nwtx\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604167 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604144    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-certs" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:37.604208 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604190    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-credential-socket\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604257 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604230    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-socket\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604257 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604246    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-data" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:37.604379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604317    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65162b68-577e-4215-a40d-d6232717b104-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65162b68-577e-4215-a40d-d6232717b104-istio-podinfo\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604342    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-istio-token\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-istio-token\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604370    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65162b68-577e-4215-a40d-d6232717b104-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65162b68-577e-4215-a40d-d6232717b104-istiod-ca-cert\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604543 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604412    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-envoy\") pod \"65162b68-577e-4215-a40d-d6232717b104\" (UID: \"65162b68-577e-4215-a40d-d6232717b104\") "
Oct 30 13:18:37.604543 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604412    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-credential-socket" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:37.604543 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604513    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-socket" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:37.604751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.604723    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65162b68-577e-4215-a40d-d6232717b104-istiod-ca-cert" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:37.605139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.605050    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.605256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.605221    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.605256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.605249    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65162b68-577e-4215-a40d-d6232717b104-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.605385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.605263    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.605385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.605276    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.606994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.606968    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-kube-api-access-5nwtx" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "kube-api-access-5nwtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:37.607162 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.607088    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-envoy" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:37.607550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.607530    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65162b68-577e-4215-a40d-d6232717b104-istio-podinfo" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:18:37.607726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.607702    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-istio-token" pod "65162b68-577e-4215-a40d-d6232717b104" (UID: "65162b68-577e-4215-a40d-d6232717b104"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:37.706915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.706879    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65162b68-577e-4215-a40d-d6232717b104-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.706915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.706914    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.707136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.706928    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65162b68-577e-4215-a40d-d6232717b104-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.707136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.706941    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nwtx\" (UniqueName: \"kubernetes.io/projected/65162b68-577e-4215-a40d-d6232717b104-kube-api-access-5nwtx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:37.893729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.893689    4429 generic.go:356] "Generic (PLEG): container finished" podID="65162b68-577e-4215-a40d-d6232717b104" containerID="2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544" exitCode=0
Oct 30 13:18:37.893921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.893755    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"
Oct 30 13:18:37.893921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.893821    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh" event={"ID":"65162b68-577e-4215-a40d-d6232717b104","Type":"ContainerDied","Data":"2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544"}
Oct 30 13:18:37.893921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.893872    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh" event={"ID":"65162b68-577e-4215-a40d-d6232717b104","Type":"ContainerDied","Data":"db690f9fff90db61637225efa673fe0493f6df57c13ce491a4eed6156caeacb7"}
Oct 30 13:18:37.893921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.893900    4429 scope.go:122] "RemoveContainer" containerID="2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544"
Oct 30 13:18:37.905402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.905383    4429 scope.go:122] "RemoveContainer" containerID="2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544"
Oct 30 13:18:37.905721 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:18:37.905701    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544\": container with ID starting with 2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544 not found: ID does not exist" containerID="2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544"
Oct 30 13:18:37.905777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.905732    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544"} err="failed to get container status \"2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544\": rpc error: code = NotFound desc = could not find container \"2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544\": container with ID starting with 2f270c74b3c6d986243a624f03e87216e3652ee707217b59f03ff3f60e4a8544 not found: ID does not exist"
Oct 30 13:18:37.981230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.981147    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"]
Oct 30 13:18:37.984370 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:37.984343    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-f1af9c-openshift-default-695c446b69-w72kh"]
Oct 30 13:18:38.091579 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:38.091545    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65162b68-577e-4215-a40d-d6232717b104" path="/var/lib/kubelet/pods/65162b68-577e-4215-a40d-d6232717b104/volumes"
Oct 30 13:18:39.222231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.222196    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"]
Oct 30 13:18:39.243155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.243121    4429 cpu_manager.go:414] "containerMap: removing container" podUID="65162b68-577e-4215-a40d-d6232717b104" containerName="istio-proxy"
Oct 30 13:18:39.243155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.243151    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65162b68-577e-4215-a40d-d6232717b104" containerName="istio-proxy"
Oct 30 13:18:39.243433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.243415    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="65162b68-577e-4215-a40d-d6232717b104" containerName="istio-proxy"
Oct 30 13:18:39.288311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.271074    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.292108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.292065    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"]
Oct 30 13:18:39.331607 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331575    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d920225c-2d94-4464-98af-fa12dc2428fd-istio-podinfo\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.331797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331698    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-socket\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.331797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331732    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d920225c-2d94-4464-98af-fa12dc2428fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.331797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331764    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-envoy\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.331797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331789    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-data\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.332002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331839    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-istio-token\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.332002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331901    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-credential-socket\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.332002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331922    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mphp\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-kube-api-access-9mphp\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.332002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.331958    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-certs\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.365359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.354757    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-451a21-openshift-default-dockercfg-hccqr\""
Oct 30 13:18:39.432871 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.432836    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-istio-token\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.432922    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-credential-socket\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.432944    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mphp\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-kube-api-access-9mphp\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.432980    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-certs\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433052    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d920225c-2d94-4464-98af-fa12dc2428fd-istio-podinfo\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433176    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-socket\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433227    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d920225c-2d94-4464-98af-fa12dc2428fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433274    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-envoy\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433329    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-data\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433516 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433399    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-credential-socket\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433516 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433418    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-certs\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433649    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-socket\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433671    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-data\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.433957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.433934    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d920225c-2d94-4464-98af-fa12dc2428fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.435869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.435844    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-envoy\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.435988 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.435972    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d920225c-2d94-4464-98af-fa12dc2428fd-istio-podinfo\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.478456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.478386    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-istio-token\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.481496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.481470    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mphp\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-kube-api-access-9mphp\") pod \"e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") " pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.618770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.618742    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:39.979321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.979271    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"]
Oct 30 13:18:39.980511 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:18:39.980485    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd920225c_2d94_4464_98af_fa12dc2428fd.slice/crio-f0b24e130136e6ed881ac64b17fefe6a8c4c011ce2c764f707c89e5e34705126 WatchSource:0}: Error finding container f0b24e130136e6ed881ac64b17fefe6a8c4c011ce2c764f707c89e5e34705126: Status 404 returned error can't find the container with id f0b24e130136e6ed881ac64b17fefe6a8c4c011ce2c764f707c89e5e34705126
Oct 30 13:18:39.983046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.983013    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:39.983138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.983079    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:39.983138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:39.983108    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:40.235263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:40.235183    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"]
Oct 30 13:18:40.235685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:40.235446    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy" containerID="cri-o://f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2" gracePeriod=30
Oct 30 13:18:40.917504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:40.917468    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" event={"ID":"d920225c-2d94-4464-98af-fa12dc2428fd","Type":"ContainerStarted","Data":"d85949087494873ab4cd8bacb282aedb466668f17aa5e1368437d8f926f9e800"}
Oct 30 13:18:40.919651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:40.917528    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" event={"ID":"d920225c-2d94-4464-98af-fa12dc2428fd","Type":"ContainerStarted","Data":"f0b24e130136e6ed881ac64b17fefe6a8c4c011ce2c764f707c89e5e34705126"}
Oct 30 13:18:40.982560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:40.982512    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" podStartSLOduration=1.982497599 podStartE2EDuration="1.982497599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:40.978766177 +0000 UTC m=+2779.034260004" watchObservedRunningTime="2026-10-30 13:18:40.982497599 +0000 UTC m=+2779.037991422"
Oct 30 13:18:41.378465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.378424    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"]
Oct 30 13:18:41.383538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.383514    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:41.385903 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.385880    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"]
Oct 30 13:18:41.397609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.397583    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67f2j\" (UniqueName: \"kubernetes.io/projected/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-kube-api-access-67f2j\") pod \"payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv\" (UID: \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\") " pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:41.397919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.397899    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv\" (UID: \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\") " pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:41.402131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.402106    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-182377\""
Oct 30 13:18:41.402809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.402788    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-182377-dockercfg-jsswm\""
Oct 30 13:18:41.499805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.499765    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv\" (UID: \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\") " pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:41.500812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.500780    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67f2j\" (UniqueName: \"kubernetes.io/projected/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-kube-api-access-67f2j\") pod \"payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv\" (UID: \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\") " pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:41.502064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.502037    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv\" (UID: \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\") " pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:41.552309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.549080    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67f2j\" (UniqueName: \"kubernetes.io/projected/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-kube-api-access-67f2j\") pod \"payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv\" (UID: \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\") " pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:41.619567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.619534    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:41.629312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.629219    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:41.702036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.702008    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:41.925052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.924980    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:41.925850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:41.925833    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:18:42.030089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:42.030065    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"]
Oct 30 13:18:42.030815 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:18:42.030787    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf09fc6_0e8b_43fa_a2af_70a395bdf0a3.slice/crio-102ae9d3f16a730f6c66dcd46863e398c1a3d0d4a0cb29d712ca63df8af9c035 WatchSource:0}: Error finding container 102ae9d3f16a730f6c66dcd46863e398c1a3d0d4a0cb29d712ca63df8af9c035: Status 404 returned error can't find the container with id 102ae9d3f16a730f6c66dcd46863e398c1a3d0d4a0cb29d712ca63df8af9c035
Oct 30 13:18:42.371663 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:18:42.371625    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:18:42.933762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:42.933720    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv" event={"ID":"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3","Type":"ContainerStarted","Data":"2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa"}
Oct 30 13:18:42.934161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:42.933828    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:42.934161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:42.933853    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv" event={"ID":"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3","Type":"ContainerStarted","Data":"102ae9d3f16a730f6c66dcd46863e398c1a3d0d4a0cb29d712ca63df8af9c035"}
Oct 30 13:18:42.978925 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:42.978880    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv" podStartSLOduration=1.9788686210000002 podStartE2EDuration="1.978868621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:42.978259842 +0000 UTC m=+2781.033753665" watchObservedRunningTime="2026-10-30 13:18:42.978868621 +0000 UTC m=+2781.034362445"
Oct 30 13:18:45.915973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.915946    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:18:45.958753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.958724    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-envoy\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.958986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.958835    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-credential-socket\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.958986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.958879    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-token\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-token\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.958986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.958904    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-socket\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.959161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.958994    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-data\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.959161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.959019    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-certs\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.959161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.959144    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-podinfo\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.959374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.959196    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3069c919-876a-4e81-89c2-05a39f1b4f51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3069c919-876a-4e81-89c2-05a39f1b4f51-istiod-ca-cert\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.959374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.959225    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-kube-api-access-w5llq\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-kube-api-access-w5llq\") pod \"3069c919-876a-4e81-89c2-05a39f1b4f51\" (UID: \"3069c919-876a-4e81-89c2-05a39f1b4f51\") "
Oct 30 13:18:45.959541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.959513    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-data" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:45.959718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.959693    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-credential-socket" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:45.960037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.960014    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3069c919-876a-4e81-89c2-05a39f1b4f51-istiod-ca-cert" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:45.960239 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.960164    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:45.960239 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.960192    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:45.960540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.960519    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-certs" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:45.960659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.960555    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-socket" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:45.963748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.963717    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-podinfo" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:18:45.963869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.963746    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-token" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:45.963869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.963816    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-envoy" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:45.963869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.963817    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-kube-api-access-w5llq" pod "3069c919-876a-4e81-89c2-05a39f1b4f51" (UID: "3069c919-876a-4e81-89c2-05a39f1b4f51"). InnerVolumeSpecName "kube-api-access-w5llq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:45.968625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.968599    4429 generic.go:356] "Generic (PLEG): container finished" podID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerID="f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2" exitCode=0
Oct 30 13:18:45.968744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.968654    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"
Oct 30 13:18:45.968744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.968687    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" event={"ID":"3069c919-876a-4e81-89c2-05a39f1b4f51","Type":"ContainerDied","Data":"f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2"}
Oct 30 13:18:45.968849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.968754    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r" event={"ID":"3069c919-876a-4e81-89c2-05a39f1b4f51","Type":"ContainerDied","Data":"7b9c61a9c76465175e4e04a5df8fb5b933cb357935620000850c60b90fb53320"}
Oct 30 13:18:45.968849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:45.968784    4429 scope.go:122] "RemoveContainer" containerID="f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2"
Oct 30 13:18:46.045791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.045765    4429 scope.go:122] "RemoveContainer" containerID="f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2"
Oct 30 13:18:46.046221 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:18:46.046194    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2\": container with ID starting with f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2 not found: ID does not exist" containerID="f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2"
Oct 30 13:18:46.046336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.046234    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2"} err="failed to get container status \"f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2\": rpc error: code = NotFound desc = could not find container \"f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2\": container with ID starting with f240de206c40ef6b28232d3a16f930e7f9b1cbd0cb47855e59d0c3bbb2a5cbe2 not found: ID does not exist"
Oct 30 13:18:46.061456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.061423    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:46.061456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.061451    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:46.061456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.061464    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:46.061700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.061473    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:46.061700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.061480    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3069c919-876a-4e81-89c2-05a39f1b4f51-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:46.061700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.061488    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5llq\" (UniqueName: \"kubernetes.io/projected/3069c919-876a-4e81-89c2-05a39f1b4f51-kube-api-access-w5llq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:46.061700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.061496    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3069c919-876a-4e81-89c2-05a39f1b4f51-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:18:46.090544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.090516    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"]
Oct 30 13:18:46.095330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.095239    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-vgv9r"]
Oct 30 13:18:46.978225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.978194    4429 generic.go:356] "Generic (PLEG): container finished" podID="0f6dd987-feca-427c-8f9b-724c899c2678" containerID="7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb" exitCode=0
Oct 30 13:18:46.978683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.978275    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" event={"ID":"0f6dd987-feca-427c-8f9b-724c899c2678","Type":"ContainerDied","Data":"7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb"}
Oct 30 13:18:46.978852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.978829    4429 scope.go:122] "RemoveContainer" containerID="7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb"
Oct 30 13:18:46.980851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.980818    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:46.980999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.980910    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:46.980999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:46.980956    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:47.990938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:47.990892    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" event={"ID":"0f6dd987-feca-427c-8f9b-724c899c2678","Type":"ContainerStarted","Data":"ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950"}
Oct 30 13:18:48.080302 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:48.080255    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" path="/var/lib/kubelet/pods/3069c919-876a-4e81-89c2-05a39f1b4f51/volumes"
Oct 30 13:18:48.837426 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:48.837394    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:18:48.837695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:48.837670    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:18:48.837802 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:48.837727    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:18:49.837811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:49.837778    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:18:49.838193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:49.837837    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:18:50.837538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:50.837509    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:18:50.837752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:50.837562    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:18:51.837884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:51.837846    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:18:51.838381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:51.837918    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:18:52.837793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:52.837758    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 13:18:52.837977 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:52.837809    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 13:18:53.843272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:53.843241    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:18:53.843703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:53.843544    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:18:53.844259 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:53.844244    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:18:53.950128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:53.950100    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:18:55.575753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:55.575719    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"]
Oct 30 13:18:55.576212 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:55.575977    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy" containerID="cri-o://07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596" gracePeriod=30
Oct 30 13:18:58.986780 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:18:58.986745    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"]
Oct 30 13:19:00.856984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.856893    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 13:19:00.912772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.912749    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-credential-socket\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.912930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.912789    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-podinfo\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.912930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.912835    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-socket\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.912930 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.912874    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-kube-api-access-vj4wc\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-kube-api-access-vj4wc\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.913118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.912934    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-token\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-token\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.913118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.912969    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-certs\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.913118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.913025    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istiod-ca-cert\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.913118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.913039    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-credential-socket" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.913379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.913140    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-envoy\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.913379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.913154    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-socket" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.913379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.913195    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-data\") pod \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\" (UID: \"9b4812a4-0f77-4113-855f-f0adb8ad1ea2\") "
Oct 30 13:19:00.914916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.914846    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istiod-ca-cert" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:00.914916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.913337    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-certs" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.915640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.915518    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-data" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:00.915640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.915621    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.915812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.915644    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:00.917409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.917378    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-podinfo" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:19:00.918090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.918035    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-kube-api-access-vj4wc" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "kube-api-access-vj4wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:00.920379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.919058    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-token" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:00.928686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:00.928636    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-envoy" pod "9b4812a4-0f77-4113-855f-f0adb8ad1ea2" (UID: "9b4812a4-0f77-4113-855f-f0adb8ad1ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:01.016885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.016820    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:01.016885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.016861    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4wc\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-kube-api-access-vj4wc\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:01.016885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.016880    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:01.017188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.016896    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:01.017188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.016910    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:01.017188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.016923    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:01.017188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.016936    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4812a4-0f77-4113-855f-f0adb8ad1ea2-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:01.102019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.101987    4429 generic.go:356] "Generic (PLEG): container finished" podID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerID="07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596" exitCode=0
Oct 30 13:19:01.102185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.102074    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"
Oct 30 13:19:01.102185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.102081    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" event={"ID":"9b4812a4-0f77-4113-855f-f0adb8ad1ea2","Type":"ContainerDied","Data":"07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596"}
Oct 30 13:19:01.102185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.102158    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq" event={"ID":"9b4812a4-0f77-4113-855f-f0adb8ad1ea2","Type":"ContainerDied","Data":"c3076c199420c9e6f63136acb936bed7d643bd6f03754c7d77b1795b3f6b752f"}
Oct 30 13:19:01.102185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.102175    4429 scope.go:122] "RemoveContainer" containerID="07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596"
Oct 30 13:19:01.114134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.114116    4429 scope.go:122] "RemoveContainer" containerID="07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596"
Oct 30 13:19:01.114450 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:01.114432    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596\": container with ID starting with 07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596 not found: ID does not exist" containerID="07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596"
Oct 30 13:19:01.114526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.114463    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596"} err="failed to get container status \"07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596\": rpc error: code = NotFound desc = could not find container \"07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596\": container with ID starting with 07176a2231fa68f0f29e0b885626a895f95a4ff51e856454eede4174ad391596 not found: ID does not exist"
Oct 30 13:19:01.161179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.161091    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"]
Oct 30 13:19:01.166146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:01.166122    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-551dd1-openshift-default-8479fbc77f-7lndq"]
Oct 30 13:19:02.077463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:02.077433    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" path="/var/lib/kubelet/pods/9b4812a4-0f77-4113-855f-f0adb8ad1ea2/volumes"
Oct 30 13:19:08.910633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:08.910599    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"]
Oct 30 13:19:12.373005 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:12.372963    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:19:12.721486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.721455    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"]
Oct 30 13:19:12.722544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.722516    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy"
Oct 30 13:19:12.722544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.722539    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy"
Oct 30 13:19:12.722765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.722557    4429 cpu_manager.go:414] "containerMap: removing container" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy"
Oct 30 13:19:12.722765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.722565    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy"
Oct 30 13:19:12.722765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.722703    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4812a4-0f77-4113-855f-f0adb8ad1ea2" containerName="istio-proxy"
Oct 30 13:19:12.722765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.722715    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="3069c919-876a-4e81-89c2-05a39f1b4f51" containerName="istio-proxy"
Oct 30 13:19:12.728225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.728194    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.752839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.752813    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"]
Oct 30 13:19:12.798821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.798791    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-451a21-serving-cert\""
Oct 30 13:19:12.875807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.875769    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdst\" (UniqueName: \"kubernetes.io/projected/3c196508-b0b5-44eb-8be9-190309a6184e-kube-api-access-djdst\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.875986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.875820    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.875986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.875900    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.875986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.875974    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c196508-b0b5-44eb-8be9-190309a6184e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.977710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.977624    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdst\" (UniqueName: \"kubernetes.io/projected/3c196508-b0b5-44eb-8be9-190309a6184e-kube-api-access-djdst\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.977710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.977674    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.977710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.977699    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.977955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.977735    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c196508-b0b5-44eb-8be9-190309a6184e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.978471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.978439    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c196508-b0b5-44eb-8be9-190309a6184e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.980297 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.980267    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:12.980375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:12.980265    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:13.003039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:13.003013    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdst\" (UniqueName: \"kubernetes.io/projected/3c196508-b0b5-44eb-8be9-190309a6184e-kube-api-access-djdst\") pod \"maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:13.046678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:13.046646    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:13.372055 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:19:13.372027    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c196508_b0b5_44eb_8be9_190309a6184e.slice/crio-caadb2f7c45d3def5e5eab13a477cef9024d79f28250e1208e2a2f86dca54bd2 WatchSource:0}: Error finding container caadb2f7c45d3def5e5eab13a477cef9024d79f28250e1208e2a2f86dca54bd2: Status 404 returned error can't find the container with id caadb2f7c45d3def5e5eab13a477cef9024d79f28250e1208e2a2f86dca54bd2
Oct 30 13:19:13.372055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:13.372030    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"]
Oct 30 13:19:14.016330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.011547    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw" podUID="fc0f79fe-2234-47ab-bee0-7301c97a6104" containerName="main" containerID="cri-o://7131a3b837859cbd39221424f762be3aa494d53d1d1ca0e7fe1bcfdc306d6743" gracePeriod=45
Oct 30 13:19:14.202304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.202252    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m" event={"ID":"3c196508-b0b5-44eb-8be9-190309a6184e","Type":"ContainerStarted","Data":"dab3f4768e231ef324d3e4e8adbf5870d94ddb17706f0a80421cd868ccc5cf12"}
Oct 30 13:19:14.202508 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.202331    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m" event={"ID":"3c196508-b0b5-44eb-8be9-190309a6184e","Type":"ContainerStarted","Data":"caadb2f7c45d3def5e5eab13a477cef9024d79f28250e1208e2a2f86dca54bd2"}
Oct 30 13:19:14.202508 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.202460    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:14.204090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.204061    4429 generic.go:356] "Generic (PLEG): container finished" podID="fc0f79fe-2234-47ab-bee0-7301c97a6104" containerID="7131a3b837859cbd39221424f762be3aa494d53d1d1ca0e7fe1bcfdc306d6743" exitCode=0
Oct 30 13:19:14.204224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.204120    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw" event={"ID":"fc0f79fe-2234-47ab-bee0-7301c97a6104","Type":"ContainerDied","Data":"7131a3b837859cbd39221424f762be3aa494d53d1d1ca0e7fe1bcfdc306d6743"}
Oct 30 13:19:14.277273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.277157    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m" podStartSLOduration=1.882681583 podStartE2EDuration="2.277138141s" totalImagesPullingTime="394.456558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:14.273922551 +0000 UTC m=+2812.329416393" watchObservedRunningTime="2026-10-30 13:19:14.277138141 +0000 UTC m=+2812.332631966"
Oct 30 13:19:14.308271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.308244    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:19:14.397180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.397143    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-dshm\") pod \"fc0f79fe-2234-47ab-bee0-7301c97a6104\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") "
Oct 30 13:19:14.397394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.397213    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0f79fe-2234-47ab-bee0-7301c97a6104-kube-api-access-p8q56\" (UniqueName: \"kubernetes.io/projected/fc0f79fe-2234-47ab-bee0-7301c97a6104-kube-api-access-p8q56\") pod \"fc0f79fe-2234-47ab-bee0-7301c97a6104\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") "
Oct 30 13:19:14.397394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.397261    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-model-cache\") pod \"fc0f79fe-2234-47ab-bee0-7301c97a6104\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") "
Oct 30 13:19:14.397394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.397306    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-tmp-dir\") pod \"fc0f79fe-2234-47ab-bee0-7301c97a6104\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") "
Oct 30 13:19:14.397394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.397362    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-home\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-home\") pod \"fc0f79fe-2234-47ab-bee0-7301c97a6104\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") "
Oct 30 13:19:14.397632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.397570    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc0f79fe-2234-47ab-bee0-7301c97a6104-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0f79fe-2234-47ab-bee0-7301c97a6104-tls-certs\") pod \"fc0f79fe-2234-47ab-bee0-7301c97a6104\" (UID: \"fc0f79fe-2234-47ab-bee0-7301c97a6104\") "
Oct 30 13:19:14.397632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.397580    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-model-cache" pod "fc0f79fe-2234-47ab-bee0-7301c97a6104" (UID: "fc0f79fe-2234-47ab-bee0-7301c97a6104"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:14.397805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.397779    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-home" pod "fc0f79fe-2234-47ab-bee0-7301c97a6104" (UID: "fc0f79fe-2234-47ab-bee0-7301c97a6104"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:14.398219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.398150    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:14.398219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.398179    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:14.398611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.398579    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-tmp-dir" pod "fc0f79fe-2234-47ab-bee0-7301c97a6104" (UID: "fc0f79fe-2234-47ab-bee0-7301c97a6104"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:14.400146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.400108    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0f79fe-2234-47ab-bee0-7301c97a6104-tls-certs" pod "fc0f79fe-2234-47ab-bee0-7301c97a6104" (UID: "fc0f79fe-2234-47ab-bee0-7301c97a6104"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:14.400300 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.400270    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-dshm" pod "fc0f79fe-2234-47ab-bee0-7301c97a6104" (UID: "fc0f79fe-2234-47ab-bee0-7301c97a6104"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:14.400372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.400316    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0f79fe-2234-47ab-bee0-7301c97a6104-kube-api-access-p8q56" pod "fc0f79fe-2234-47ab-bee0-7301c97a6104" (UID: "fc0f79fe-2234-47ab-bee0-7301c97a6104"). InnerVolumeSpecName "kube-api-access-p8q56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:14.500527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.500484    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0f79fe-2234-47ab-bee0-7301c97a6104-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:14.500527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.500521    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:14.500527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.500537    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8q56\" (UniqueName: \"kubernetes.io/projected/fc0f79fe-2234-47ab-bee0-7301c97a6104-kube-api-access-p8q56\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:14.500527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:14.500557    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0f79fe-2234-47ab-bee0-7301c97a6104-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:15.213338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:15.213309    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"
Oct 30 13:19:15.213338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:15.213329    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw" event={"ID":"fc0f79fe-2234-47ab-bee0-7301c97a6104","Type":"ContainerDied","Data":"8531e157c4da3ecf78d01436617bbc52789edd115ef9192177fe94cbbbc8e759"}
Oct 30 13:19:15.213849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:15.213372    4429 scope.go:122] "RemoveContainer" containerID="7131a3b837859cbd39221424f762be3aa494d53d1d1ca0e7fe1bcfdc306d6743"
Oct 30 13:19:15.322189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:15.322162    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"]
Oct 30 13:19:15.338151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:15.338127    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/auth-test-model-517a25-kserve-8458d988fc-64lmw"]
Oct 30 13:19:16.076659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.076624    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0f79fe-2234-47ab-bee0-7301c97a6104" path="/var/lib/kubelet/pods/fc0f79fe-2234-47ab-bee0-7301c97a6104/volumes"
Oct 30 13:19:16.842824 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.842794    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"]
Oct 30 13:19:16.843525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.843507    4429 cpu_manager.go:414] "containerMap: removing container" podUID="fc0f79fe-2234-47ab-bee0-7301c97a6104" containerName="main"
Oct 30 13:19:16.843570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.843527    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0f79fe-2234-47ab-bee0-7301c97a6104" containerName="main"
Oct 30 13:19:16.843641 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.843632    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0f79fe-2234-47ab-bee0-7301c97a6104" containerName="main"
Oct 30 13:19:16.848725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.848703    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:16.851460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.851432    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"]
Oct 30 13:19:16.855798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.855779    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:16.865403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.865377    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1957dc\""
Oct 30 13:19:16.879083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.879053    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"]
Oct 30 13:19:16.890013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.889985    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"]
Oct 30 13:19:16.907462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:16.907436    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1957dc-dockercfg-9gg4l\""
Oct 30 13:19:17.035561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.034791    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cwh5\" (UniqueName: \"kubernetes.io/projected/c74aba3a-269e-4d9c-88a3-4b224d852c36-kube-api-access-8cwh5\") pod \"payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl\" (UID: \"c74aba3a-269e-4d9c-88a3-4b224d852c36\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:17.035561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.035134    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74aba3a-269e-4d9c-88a3-4b224d852c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl\" (UID: \"c74aba3a-269e-4d9c-88a3-4b224d852c36\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:17.035561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.035262    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfjw\" (UniqueName: \"kubernetes.io/projected/1ca25a94-48bd-4048-9955-09b137fbefe4-kube-api-access-dmfjw\") pod \"payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4\" (UID: \"1ca25a94-48bd-4048-9955-09b137fbefe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:17.035561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.035341    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca25a94-48bd-4048-9955-09b137fbefe4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4\" (UID: \"1ca25a94-48bd-4048-9955-09b137fbefe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:17.136943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.136867    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74aba3a-269e-4d9c-88a3-4b224d852c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl\" (UID: \"c74aba3a-269e-4d9c-88a3-4b224d852c36\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:17.137107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.136948    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfjw\" (UniqueName: \"kubernetes.io/projected/1ca25a94-48bd-4048-9955-09b137fbefe4-kube-api-access-dmfjw\") pod \"payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4\" (UID: \"1ca25a94-48bd-4048-9955-09b137fbefe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:17.137107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.136983    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca25a94-48bd-4048-9955-09b137fbefe4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4\" (UID: \"1ca25a94-48bd-4048-9955-09b137fbefe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:17.137107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.137061    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cwh5\" (UniqueName: \"kubernetes.io/projected/c74aba3a-269e-4d9c-88a3-4b224d852c36-kube-api-access-8cwh5\") pod \"payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl\" (UID: \"c74aba3a-269e-4d9c-88a3-4b224d852c36\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:17.137657 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.137632    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74aba3a-269e-4d9c-88a3-4b224d852c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl\" (UID: \"c74aba3a-269e-4d9c-88a3-4b224d852c36\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:17.137828 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.137808    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca25a94-48bd-4048-9955-09b137fbefe4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4\" (UID: \"1ca25a94-48bd-4048-9955-09b137fbefe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:17.162135 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.162108    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cwh5\" (UniqueName: \"kubernetes.io/projected/c74aba3a-269e-4d9c-88a3-4b224d852c36-kube-api-access-8cwh5\") pod \"payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl\" (UID: \"c74aba3a-269e-4d9c-88a3-4b224d852c36\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:17.166188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.166166    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfjw\" (UniqueName: \"kubernetes.io/projected/1ca25a94-48bd-4048-9955-09b137fbefe4-kube-api-access-dmfjw\") pod \"payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4\" (UID: \"1ca25a94-48bd-4048-9955-09b137fbefe4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:17.166400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.166371    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:17.177112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.177090    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:17.506383 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:19:17.506348    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74aba3a_269e_4d9c_88a3_4b224d852c36.slice/crio-9d401768ecf25215e33d7ae39d5dcf02bfcea088c37e3eec23bdafa2da22c4e6 WatchSource:0}: Error finding container 9d401768ecf25215e33d7ae39d5dcf02bfcea088c37e3eec23bdafa2da22c4e6: Status 404 returned error can't find the container with id 9d401768ecf25215e33d7ae39d5dcf02bfcea088c37e3eec23bdafa2da22c4e6
Oct 30 13:19:17.507573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.507548    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"]
Oct 30 13:19:17.516778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:17.516755    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"]
Oct 30 13:19:18.243862 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:18.243822    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4" event={"ID":"1ca25a94-48bd-4048-9955-09b137fbefe4","Type":"ContainerStarted","Data":"06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc"}
Oct 30 13:19:18.244359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:18.243893    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:18.244359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:18.243917    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4" event={"ID":"1ca25a94-48bd-4048-9955-09b137fbefe4","Type":"ContainerStarted","Data":"7c2f3f0172485487f21528c5db856f4112f70eca835c7f49e8bd56a8928cbcf4"}
Oct 30 13:19:18.245487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:18.245455    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl" event={"ID":"c74aba3a-269e-4d9c-88a3-4b224d852c36","Type":"ContainerStarted","Data":"0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94"}
Oct 30 13:19:18.245589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:18.245508    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:18.245589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:18.245535    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl" event={"ID":"c74aba3a-269e-4d9c-88a3-4b224d852c36","Type":"ContainerStarted","Data":"9d401768ecf25215e33d7ae39d5dcf02bfcea088c37e3eec23bdafa2da22c4e6"}
Oct 30 13:19:18.298401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:18.298358    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4" podStartSLOduration=2.298345317 podStartE2EDuration="2.298345317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:18.297856054 +0000 UTC m=+2816.353349903" watchObservedRunningTime="2026-10-30 13:19:18.298345317 +0000 UTC m=+2816.353839142"
Oct 30 13:19:18.357322 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:18.357244    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl" podStartSLOduration=2.357226348 podStartE2EDuration="2.357226348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:18.354346095 +0000 UTC m=+2816.409839939" watchObservedRunningTime="2026-10-30 13:19:18.357226348 +0000 UTC m=+2816.412720176"
Oct 30 13:19:19.376870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.376836    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11532-b7a498"]
Oct 30 13:19:19.381390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.381374    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:19.401346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.401320    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-b7a498"]
Oct 30 13:19:19.410516 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.410486    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ljkrn\""
Oct 30 13:19:19.476853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.476820    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhsw\" (UniqueName: \"kubernetes.io/projected/24c04f54-e635-4597-b7a6-c6a7f9ee09e3-kube-api-access-bqhsw\") pod \"test-curl-11532-b7a498\" (UID: \"24c04f54-e635-4597-b7a6-c6a7f9ee09e3\") " pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:19.578492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.578457    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhsw\" (UniqueName: \"kubernetes.io/projected/24c04f54-e635-4597-b7a6-c6a7f9ee09e3-kube-api-access-bqhsw\") pod \"test-curl-11532-b7a498\" (UID: \"24c04f54-e635-4597-b7a6-c6a7f9ee09e3\") " pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:19.604058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.604034    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhsw\" (UniqueName: \"kubernetes.io/projected/24c04f54-e635-4597-b7a6-c6a7f9ee09e3-kube-api-access-bqhsw\") pod \"test-curl-11532-b7a498\" (UID: \"24c04f54-e635-4597-b7a6-c6a7f9ee09e3\") " pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:19.695169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.695142    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:19.996521 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:19:19.996490    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24c04f54_e635_4597_b7a6_c6a7f9ee09e3.slice/crio-95ff391bf88cec4b05e1a928b08d82c981a4e4e8ef80e35f129e0449c637e897 WatchSource:0}: Error finding container 95ff391bf88cec4b05e1a928b08d82c981a4e4e8ef80e35f129e0449c637e897: Status 404 returned error can't find the container with id 95ff391bf88cec4b05e1a928b08d82c981a4e4e8ef80e35f129e0449c637e897
Oct 30 13:19:19.997419 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:19.997327    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-b7a498"]
Oct 30 13:19:20.219173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:20.219147    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:19:20.259997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:20.259915    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-b7a498" event={"ID":"24c04f54-e635-4597-b7a6-c6a7f9ee09e3","Type":"ContainerStarted","Data":"8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f"}
Oct 30 13:19:20.259997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:20.259958    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-b7a498" event={"ID":"24c04f54-e635-4597-b7a6-c6a7f9ee09e3","Type":"ContainerStarted","Data":"95ff391bf88cec4b05e1a928b08d82c981a4e4e8ef80e35f129e0449c637e897"}
Oct 30 13:19:20.330610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:20.330557    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11532-b7a498" podStartSLOduration=1.33054309 podStartE2EDuration="1.33054309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:20.329260317 +0000 UTC m=+2818.384754141" watchObservedRunningTime="2026-10-30 13:19:20.33054309 +0000 UTC m=+2818.386036913"
Oct 30 13:19:21.202847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.202810    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11532-b7a498"]
Oct 30 13:19:21.207567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.207537    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11532-b7a498"]
Oct 30 13:19:21.361920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.361889    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11532-7b026e"]
Oct 30 13:19:21.362600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.362584    4429 cpu_manager.go:414] "containerMap: removing container" podUID="24c04f54-e635-4597-b7a6-c6a7f9ee09e3" containerName="test-curl-11532-b7a498"
Oct 30 13:19:21.362659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.362601    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c04f54-e635-4597-b7a6-c6a7f9ee09e3" containerName="test-curl-11532-b7a498"
Oct 30 13:19:21.362699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.362690    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c04f54-e635-4597-b7a6-c6a7f9ee09e3" containerName="test-curl-11532-b7a498"
Oct 30 13:19:21.366222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.366202    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-7b026e"
Oct 30 13:19:21.385564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.385539    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-7b026e"]
Oct 30 13:19:21.400928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.400905    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4rf\" (UniqueName: \"kubernetes.io/projected/c8690d9b-7b32-48b6-88bb-1e22d2d0124d-kube-api-access-np4rf\") pod \"test-curl-11532-7b026e\" (UID: \"c8690d9b-7b32-48b6-88bb-1e22d2d0124d\") " pod="opendatahub/test-curl-11532-7b026e"
Oct 30 13:19:21.502375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.502305    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4rf\" (UniqueName: \"kubernetes.io/projected/c8690d9b-7b32-48b6-88bb-1e22d2d0124d-kube-api-access-np4rf\") pod \"test-curl-11532-7b026e\" (UID: \"c8690d9b-7b32-48b6-88bb-1e22d2d0124d\") " pod="opendatahub/test-curl-11532-7b026e"
Oct 30 13:19:21.534974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.534943    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4rf\" (UniqueName: \"kubernetes.io/projected/c8690d9b-7b32-48b6-88bb-1e22d2d0124d-kube-api-access-np4rf\") pod \"test-curl-11532-7b026e\" (UID: \"c8690d9b-7b32-48b6-88bb-1e22d2d0124d\") " pod="opendatahub/test-curl-11532-7b026e"
Oct 30 13:19:21.678389 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.678351    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-7b026e"
Oct 30 13:19:21.998181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:21.998060    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-7b026e"]
Oct 30 13:19:22.276900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:22.276806    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-7b026e" event={"ID":"c8690d9b-7b32-48b6-88bb-1e22d2d0124d","Type":"ContainerStarted","Data":"7cd1af61756a37409752823e6da237f283603384965ebd8e800194e4403d6f1c"}
Oct 30 13:19:22.276900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:22.276858    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-7b026e" event={"ID":"c8690d9b-7b32-48b6-88bb-1e22d2d0124d","Type":"ContainerStarted","Data":"88a6a023f4dfbaaa34ed72c6c788c6da471c7f3bf7e1349758eff7f464996d1c"}
Oct 30 13:19:22.277429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:22.277091    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11532-b7a498" podUID="24c04f54-e635-4597-b7a6-c6a7f9ee09e3" containerName="test-curl-11532-b7a498" containerID="cri-o://8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f" gracePeriod=2
Oct 30 13:19:22.339192 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:22.339133    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11532-7b026e" podStartSLOduration=1.339118764 podStartE2EDuration="1.339118764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:22.33785723 +0000 UTC m=+2820.393351038" watchObservedRunningTime="2026-10-30 13:19:22.339118764 +0000 UTC m=+2820.394612588"
Oct 30 13:19:23.272961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.272929    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11532-7b026e"]
Oct 30 13:19:23.276077 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.276051    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11532-7b026e"]
Oct 30 13:19:23.441766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.441738    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11532-8efc6a"]
Oct 30 13:19:23.442465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.442450    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c8690d9b-7b32-48b6-88bb-1e22d2d0124d" containerName="test-curl-11532-7b026e"
Oct 30 13:19:23.442512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.442466    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8690d9b-7b32-48b6-88bb-1e22d2d0124d" containerName="test-curl-11532-7b026e"
Oct 30 13:19:23.442575 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.442569    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8690d9b-7b32-48b6-88bb-1e22d2d0124d" containerName="test-curl-11532-7b026e"
Oct 30 13:19:23.446415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.446395    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:23.463969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.463945    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-8efc6a"]
Oct 30 13:19:23.526872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.526816    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws75h\" (UniqueName: \"kubernetes.io/projected/a223e266-09d7-47e0-95e6-ce53aa23b366-kube-api-access-ws75h\") pod \"test-curl-11532-8efc6a\" (UID: \"a223e266-09d7-47e0-95e6-ce53aa23b366\") " pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:23.628211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.628171    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws75h\" (UniqueName: \"kubernetes.io/projected/a223e266-09d7-47e0-95e6-ce53aa23b366-kube-api-access-ws75h\") pod \"test-curl-11532-8efc6a\" (UID: \"a223e266-09d7-47e0-95e6-ce53aa23b366\") " pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:23.653105 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.653075    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws75h\" (UniqueName: \"kubernetes.io/projected/a223e266-09d7-47e0-95e6-ce53aa23b366-kube-api-access-ws75h\") pod \"test-curl-11532-8efc6a\" (UID: \"a223e266-09d7-47e0-95e6-ce53aa23b366\") " pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:23.759904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.759870    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:23.933252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:23.933209    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8" podUID="7a9a7b93-aaf2-4ae4-b353-168065918adf" containerName="main" containerID="cri-o://2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a" gracePeriod=45
Oct 30 13:19:24.062347 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:19:24.062317    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda223e266_09d7_47e0_95e6_ce53aa23b366.slice/crio-cc1080c86f403d3a7679146719ccead24e3cfc8bc59c30619534c1ddcee958e1 WatchSource:0}: Error finding container cc1080c86f403d3a7679146719ccead24e3cfc8bc59c30619534c1ddcee958e1: Status 404 returned error can't find the container with id cc1080c86f403d3a7679146719ccead24e3cfc8bc59c30619534c1ddcee958e1
Oct 30 13:19:24.063618 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.063593    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-8efc6a"]
Oct 30 13:19:24.192434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.192410    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:19:24.237651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.237619    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-dshm\") pod \"7a9a7b93-aaf2-4ae4-b353-168065918adf\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") "
Oct 30 13:19:24.241392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.237754    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-model-cache\") pod \"7a9a7b93-aaf2-4ae4-b353-168065918adf\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") "
Oct 30 13:19:24.241392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.237787    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9a7b93-aaf2-4ae4-b353-168065918adf-kube-api-access-spnjr\" (UniqueName: \"kubernetes.io/projected/7a9a7b93-aaf2-4ae4-b353-168065918adf-kube-api-access-spnjr\") pod \"7a9a7b93-aaf2-4ae4-b353-168065918adf\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") "
Oct 30 13:19:24.241392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.237828    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9a7b93-aaf2-4ae4-b353-168065918adf-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9a7b93-aaf2-4ae4-b353-168065918adf-tls-certs\") pod \"7a9a7b93-aaf2-4ae4-b353-168065918adf\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") "
Oct 30 13:19:24.241392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.238000    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-home\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-home\") pod \"7a9a7b93-aaf2-4ae4-b353-168065918adf\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") "
Oct 30 13:19:24.241392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.238036    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-tmp-dir\") pod \"7a9a7b93-aaf2-4ae4-b353-168065918adf\" (UID: \"7a9a7b93-aaf2-4ae4-b353-168065918adf\") "
Oct 30 13:19:24.241392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.238696    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-tmp-dir" pod "7a9a7b93-aaf2-4ae4-b353-168065918adf" (UID: "7a9a7b93-aaf2-4ae4-b353-168065918adf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:24.241392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.239623    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-model-cache" pod "7a9a7b93-aaf2-4ae4-b353-168065918adf" (UID: "7a9a7b93-aaf2-4ae4-b353-168065918adf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:24.241392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.239855    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-home" pod "7a9a7b93-aaf2-4ae4-b353-168065918adf" (UID: "7a9a7b93-aaf2-4ae4-b353-168065918adf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:24.241889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.241402    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-dshm" pod "7a9a7b93-aaf2-4ae4-b353-168065918adf" (UID: "7a9a7b93-aaf2-4ae4-b353-168065918adf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:19:24.242239 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.242212    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9a7b93-aaf2-4ae4-b353-168065918adf-tls-certs" pod "7a9a7b93-aaf2-4ae4-b353-168065918adf" (UID: "7a9a7b93-aaf2-4ae4-b353-168065918adf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:24.244724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.244698    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9a7b93-aaf2-4ae4-b353-168065918adf-kube-api-access-spnjr" pod "7a9a7b93-aaf2-4ae4-b353-168065918adf" (UID: "7a9a7b93-aaf2-4ae4-b353-168065918adf"). InnerVolumeSpecName "kube-api-access-spnjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:24.296082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.296053    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-8efc6a" event={"ID":"a223e266-09d7-47e0-95e6-ce53aa23b366","Type":"ContainerStarted","Data":"9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f"}
Oct 30 13:19:24.296220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.296096    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-8efc6a" event={"ID":"a223e266-09d7-47e0-95e6-ce53aa23b366","Type":"ContainerStarted","Data":"cc1080c86f403d3a7679146719ccead24e3cfc8bc59c30619534c1ddcee958e1"}
Oct 30 13:19:24.297459 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.297436    4429 generic.go:356] "Generic (PLEG): container finished" podID="7a9a7b93-aaf2-4ae4-b353-168065918adf" containerID="2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a" exitCode=0
Oct 30 13:19:24.297554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.297482    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"
Oct 30 13:19:24.297554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.297535    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8" event={"ID":"7a9a7b93-aaf2-4ae4-b353-168065918adf","Type":"ContainerDied","Data":"2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a"}
Oct 30 13:19:24.297629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.297562    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8" event={"ID":"7a9a7b93-aaf2-4ae4-b353-168065918adf","Type":"ContainerDied","Data":"25b887f05ee787ebb8fbce23fd064fb62bc97dc1171c1178666ab08ce22d6fa3"}
Oct 30 13:19:24.297629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.297580    4429 scope.go:122] "RemoveContainer" containerID="2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a"
Oct 30 13:19:24.297752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.297727    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11532-7b026e" podUID="c8690d9b-7b32-48b6-88bb-1e22d2d0124d" containerName="test-curl-11532-7b026e" containerID="cri-o://7cd1af61756a37409752823e6da237f283603384965ebd8e800194e4403d6f1c" gracePeriod=2
Oct 30 13:19:24.337994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.337955    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11532-8efc6a" podStartSLOduration=1.337941891 podStartE2EDuration="1.337941891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:24.337348689 +0000 UTC m=+2822.392842510" watchObservedRunningTime="2026-10-30 13:19:24.337941891 +0000 UTC m=+2822.393435715"
Oct 30 13:19:24.339404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.339382    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:24.339513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.339408    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:24.339513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.339424    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnjr\" (UniqueName: \"kubernetes.io/projected/7a9a7b93-aaf2-4ae4-b353-168065918adf-kube-api-access-spnjr\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:24.339513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.339437    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a9a7b93-aaf2-4ae4-b353-168065918adf-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:24.339513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.339451    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:24.339513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.339463    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a9a7b93-aaf2-4ae4-b353-168065918adf-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:24.352720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.352684    4429 scope.go:122] "RemoveContainer" containerID="2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a"
Oct 30 13:19:24.353059 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:24.353026    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a\": container with ID starting with 2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a not found: ID does not exist" containerID="2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a"
Oct 30 13:19:24.353131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.353071    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a"} err="failed to get container status \"2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a\": rpc error: code = NotFound desc = could not find container \"2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a\": container with ID starting with 2bd61a67f2a345569babb15bf12b4361bd409abeb3f8bb5a9487903653c64b4a not found: ID does not exist"
Oct 30 13:19:24.394609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.394572    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"]
Oct 30 13:19:24.400342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.400317    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8"]
Oct 30 13:19:24.470511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.470486    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:24.541097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.541012    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c04f54-e635-4597-b7a6-c6a7f9ee09e3-kube-api-access-bqhsw\" (UniqueName: \"kubernetes.io/projected/24c04f54-e635-4597-b7a6-c6a7f9ee09e3-kube-api-access-bqhsw\") pod \"24c04f54-e635-4597-b7a6-c6a7f9ee09e3\" (UID: \"24c04f54-e635-4597-b7a6-c6a7f9ee09e3\") "
Oct 30 13:19:24.545429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.545395    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c04f54-e635-4597-b7a6-c6a7f9ee09e3-kube-api-access-bqhsw" pod "24c04f54-e635-4597-b7a6-c6a7f9ee09e3" (UID: "24c04f54-e635-4597-b7a6-c6a7f9ee09e3"). InnerVolumeSpecName "kube-api-access-bqhsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:24.643178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:24.643099    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhsw\" (UniqueName: \"kubernetes.io/projected/24c04f54-e635-4597-b7a6-c6a7f9ee09e3-kube-api-access-bqhsw\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:25.175413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.175362    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-241ffa/auth-test-model-241ffa-kserve-84ff5ddbbf-pbkg8" podUID="7a9a7b93-aaf2-4ae4-b353-168065918adf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": context deadline exceeded"
Oct 30 13:19:25.234613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.234563    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11532-8efc6a"]
Oct 30 13:19:25.239770 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.239732    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11532-8efc6a"]
Oct 30 13:19:25.360860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.360817    4429 generic.go:356] "Generic (PLEG): container finished" podID="24c04f54-e635-4597-b7a6-c6a7f9ee09e3" containerID="8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f" exitCode=137
Oct 30 13:19:25.361056 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.360908    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:25.361056 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.360922    4429 scope.go:122] "RemoveContainer" containerID="8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f"
Oct 30 13:19:25.372267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.372240    4429 scope.go:122] "RemoveContainer" containerID="8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f"
Oct 30 13:19:25.372618 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:25.372598    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f\": container with ID starting with 8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f not found: ID does not exist" containerID="8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f"
Oct 30 13:19:25.372691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.372633    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f"} err="failed to get container status \"8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f\": rpc error: code = NotFound desc = could not find container \"8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f\": container with ID starting with 8a36254bd9d9c61fc591f5a0b1df3e24e87f7007e2f2551cc25b3f5200cc340f not found: ID does not exist"
Oct 30 13:19:25.386108 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:25.386065    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-b7a498\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="24c04f54-e635-4597-b7a6-c6a7f9ee09e3" pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:25.395309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.395248    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11532-8333e1"]
Oct 30 13:19:25.396006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.395990    4429 cpu_manager.go:414] "containerMap: removing container" podUID="7a9a7b93-aaf2-4ae4-b353-168065918adf" containerName="main"
Oct 30 13:19:25.396047 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.396009    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9a7b93-aaf2-4ae4-b353-168065918adf" containerName="main"
Oct 30 13:19:25.396079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.396050    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a223e266-09d7-47e0-95e6-ce53aa23b366" containerName="test-curl-11532-8efc6a"
Oct 30 13:19:25.396079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.396056    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a223e266-09d7-47e0-95e6-ce53aa23b366" containerName="test-curl-11532-8efc6a"
Oct 30 13:19:25.396158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.396149    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9a7b93-aaf2-4ae4-b353-168065918adf" containerName="main"
Oct 30 13:19:25.396190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.396168    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a223e266-09d7-47e0-95e6-ce53aa23b366" containerName="test-curl-11532-8efc6a"
Oct 30 13:19:25.400999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.400978    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:25.418588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.418554    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-8333e1"]
Oct 30 13:19:25.456258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.456215    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwtr\" (UniqueName: \"kubernetes.io/projected/d9770306-e191-4954-8924-2314301fab23-kube-api-access-4nwtr\") pod \"test-curl-11532-8333e1\" (UID: \"d9770306-e191-4954-8924-2314301fab23\") " pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:25.461147 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:25.461110    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-b7a498\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="24c04f54-e635-4597-b7a6-c6a7f9ee09e3" pod="opendatahub/test-curl-11532-b7a498"
Oct 30 13:19:25.558138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.558096    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwtr\" (UniqueName: \"kubernetes.io/projected/d9770306-e191-4954-8924-2314301fab23-kube-api-access-4nwtr\") pod \"test-curl-11532-8333e1\" (UID: \"d9770306-e191-4954-8924-2314301fab23\") " pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:25.583981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.583947    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwtr\" (UniqueName: \"kubernetes.io/projected/d9770306-e191-4954-8924-2314301fab23-kube-api-access-4nwtr\") pod \"test-curl-11532-8333e1\" (UID: \"d9770306-e191-4954-8924-2314301fab23\") " pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:25.714358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:25.714229    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:26.046447 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:19:26.046411    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9770306_e191_4954_8924_2314301fab23.slice/crio-98d48ee3dd72d31bf93c20cbdac0782d890faa562edd72c91ff59fe203032ff6 WatchSource:0}: Error finding container 98d48ee3dd72d31bf93c20cbdac0782d890faa562edd72c91ff59fe203032ff6: Status 404 returned error can't find the container with id 98d48ee3dd72d31bf93c20cbdac0782d890faa562edd72c91ff59fe203032ff6
Oct 30 13:19:26.046658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.046634    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11532-8333e1"]
Oct 30 13:19:26.083653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.083623    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c04f54-e635-4597-b7a6-c6a7f9ee09e3" path="/var/lib/kubelet/pods/24c04f54-e635-4597-b7a6-c6a7f9ee09e3/volumes"
Oct 30 13:19:26.084092 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.084078    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9a7b93-aaf2-4ae4-b353-168065918adf" path="/var/lib/kubelet/pods/7a9a7b93-aaf2-4ae4-b353-168065918adf/volumes"
Oct 30 13:19:26.382694 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.382491    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-8333e1" event={"ID":"d9770306-e191-4954-8924-2314301fab23","Type":"ContainerStarted","Data":"baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d"}
Oct 30 13:19:26.382694 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.382541    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11532-8333e1" event={"ID":"d9770306-e191-4954-8924-2314301fab23","Type":"ContainerStarted","Data":"98d48ee3dd72d31bf93c20cbdac0782d890faa562edd72c91ff59fe203032ff6"}
Oct 30 13:19:26.385265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.385240    4429 generic.go:356] "Generic (PLEG): container finished" podID="c8690d9b-7b32-48b6-88bb-1e22d2d0124d" containerID="7cd1af61756a37409752823e6da237f283603384965ebd8e800194e4403d6f1c" exitCode=137
Oct 30 13:19:26.385453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.385437    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11532-8efc6a" podUID="a223e266-09d7-47e0-95e6-ce53aa23b366" containerName="test-curl-11532-8efc6a" containerID="cri-o://9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f" gracePeriod=2
Oct 30 13:19:26.426395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.426341    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11532-8333e1" podStartSLOduration=1.426324773 podStartE2EDuration="1.426324773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:26.424518807 +0000 UTC m=+2824.480012631" watchObservedRunningTime="2026-10-30 13:19:26.426324773 +0000 UTC m=+2824.481818608"
Oct 30 13:19:26.543714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.543686    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-7b026e"
Oct 30 13:19:26.584812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.584716    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8690d9b-7b32-48b6-88bb-1e22d2d0124d-kube-api-access-np4rf\" (UniqueName: \"kubernetes.io/projected/c8690d9b-7b32-48b6-88bb-1e22d2d0124d-kube-api-access-np4rf\") pod \"c8690d9b-7b32-48b6-88bb-1e22d2d0124d\" (UID: \"c8690d9b-7b32-48b6-88bb-1e22d2d0124d\") "
Oct 30 13:19:26.587035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.587008    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8690d9b-7b32-48b6-88bb-1e22d2d0124d-kube-api-access-np4rf" pod "c8690d9b-7b32-48b6-88bb-1e22d2d0124d" (UID: "c8690d9b-7b32-48b6-88bb-1e22d2d0124d"). InnerVolumeSpecName "kube-api-access-np4rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:26.686756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:26.686713    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np4rf\" (UniqueName: \"kubernetes.io/projected/c8690d9b-7b32-48b6-88bb-1e22d2d0124d-kube-api-access-np4rf\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:27.410311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:27.395106    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11532-8333e1"]
Oct 30 13:19:27.410311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:27.397243    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-7b026e"
Oct 30 13:19:27.410311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:27.397313    4429 scope.go:122] "RemoveContainer" containerID="7cd1af61756a37409752823e6da237f283603384965ebd8e800194e4403d6f1c"
Oct 30 13:19:27.410311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:27.400016    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11532-8333e1"]
Oct 30 13:19:27.424102 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:27.424060    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-7b026e\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="c8690d9b-7b32-48b6-88bb-1e22d2d0124d" pod="opendatahub/test-curl-11532-7b026e"
Oct 30 13:19:28.080148 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:28.080111    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8690d9b-7b32-48b6-88bb-1e22d2d0124d" path="/var/lib/kubelet/pods/c8690d9b-7b32-48b6-88bb-1e22d2d0124d/volumes"
Oct 30 13:19:28.405352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:28.405260    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11532-8333e1" podUID="d9770306-e191-4954-8924-2314301fab23" containerName="test-curl-11532-8333e1" containerID="cri-o://baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d" gracePeriod=2
Oct 30 13:19:28.639929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:28.639860    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:28.713365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:28.713323    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a223e266-09d7-47e0-95e6-ce53aa23b366-kube-api-access-ws75h\" (UniqueName: \"kubernetes.io/projected/a223e266-09d7-47e0-95e6-ce53aa23b366-kube-api-access-ws75h\") pod \"a223e266-09d7-47e0-95e6-ce53aa23b366\" (UID: \"a223e266-09d7-47e0-95e6-ce53aa23b366\") "
Oct 30 13:19:28.715677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:28.715652    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a223e266-09d7-47e0-95e6-ce53aa23b366-kube-api-access-ws75h" pod "a223e266-09d7-47e0-95e6-ce53aa23b366" (UID: "a223e266-09d7-47e0-95e6-ce53aa23b366"). InnerVolumeSpecName "kube-api-access-ws75h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:28.815345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:28.815309    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws75h\" (UniqueName: \"kubernetes.io/projected/a223e266-09d7-47e0-95e6-ce53aa23b366-kube-api-access-ws75h\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:29.273388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.273357    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:19:29.275548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.275525    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:19:29.416367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.416327    4429 generic.go:356] "Generic (PLEG): container finished" podID="a223e266-09d7-47e0-95e6-ce53aa23b366" containerID="9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f" exitCode=137
Oct 30 13:19:29.416568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.416398    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:29.417346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.416406    4429 scope.go:122] "RemoveContainer" containerID="9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f"
Oct 30 13:19:29.428251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.428231    4429 scope.go:122] "RemoveContainer" containerID="9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f"
Oct 30 13:19:29.428624 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:29.428574    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f\": container with ID starting with 9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f not found: ID does not exist" containerID="9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f"
Oct 30 13:19:29.428674 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.428641    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f"} err="failed to get container status \"9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f\": rpc error: code = NotFound desc = could not find container \"9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f\": container with ID starting with 9f3cd9aa106333a118d2307985c7e7a705660809a6a8402ba12309f3b42e792f not found: ID does not exist"
Oct 30 13:19:29.440972 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:29.440939    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-8efc6a\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="a223e266-09d7-47e0-95e6-ce53aa23b366" pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:29.690371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.684947    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"]
Oct 30 13:19:29.690371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.686074    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d9770306-e191-4954-8924-2314301fab23" containerName="test-curl-11532-8333e1"
Oct 30 13:19:29.690371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.686097    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9770306-e191-4954-8924-2314301fab23" containerName="test-curl-11532-8333e1"
Oct 30 13:19:29.690371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.686264    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9770306-e191-4954-8924-2314301fab23" containerName="test-curl-11532-8333e1"
Oct 30 13:19:29.811702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.811637    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"]
Oct 30 13:19:29.811896 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.811771    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.831130 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:29.830747    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-8efc6a\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="a223e266-09d7-47e0-95e6-ce53aa23b366" pod="opendatahub/test-curl-11532-8efc6a"
Oct 30 13:19:29.831547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.831504    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-037a5a-serving-cert\""
Oct 30 13:19:29.873309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.857181    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhbf\" (UniqueName: \"kubernetes.io/projected/06d0f798-31cc-4548-999c-40c60f7ff934-kube-api-access-rrhbf\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.874276 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.873908    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.874585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.874563    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-metrics-certs\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.874734 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.874710    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d0f798-31cc-4548-999c-40c60f7ff934-openshift-service-ca\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.981396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.981264    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-metrics-certs\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.981643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.981621    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d0f798-31cc-4548-999c-40c60f7ff934-openshift-service-ca\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.981784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.981772    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhbf\" (UniqueName: \"kubernetes.io/projected/06d0f798-31cc-4548-999c-40c60f7ff934-kube-api-access-rrhbf\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.982233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.981892    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.982233 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:29.982053    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-serving-cert: secret "maas-api-e2e-autores-037a5a-serving-cert" not found
Oct 30 13:19:29.982233 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:29.982121    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls podName:06d0f798-31cc-4548-999c-40c60f7ff934 nodeName:}" failed. No retries permitted until 2026-10-30 13:19:30.482100756 +0000 UTC m=+2828.537594557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls") pod "maas-api-e2e-autores-037a5a-6758fbd48-fbgd9" (UID: "06d0f798-31cc-4548-999c-40c60f7ff934") : secret "maas-api-e2e-autores-037a5a-serving-cert" not found
Oct 30 13:19:29.982651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.982631    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d0f798-31cc-4548-999c-40c60f7ff934-openshift-service-ca\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:29.985702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:29.985063    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-metrics-certs\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:30.015308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.014957    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhbf\" (UniqueName: \"kubernetes.io/projected/06d0f798-31cc-4548-999c-40c60f7ff934-kube-api-access-rrhbf\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:30.084128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.084094    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a223e266-09d7-47e0-95e6-ce53aa23b366" path="/var/lib/kubelet/pods/a223e266-09d7-47e0-95e6-ce53aa23b366/volumes"
Oct 30 13:19:30.489795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.489758    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:30.492384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.492361    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls\") pod \"maas-api-e2e-autores-037a5a-6758fbd48-fbgd9\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:30.717734 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.717568    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:30.735034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.735003    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:30.793859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.793826    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9770306-e191-4954-8924-2314301fab23-kube-api-access-4nwtr\" (UniqueName: \"kubernetes.io/projected/d9770306-e191-4954-8924-2314301fab23-kube-api-access-4nwtr\") pod \"d9770306-e191-4954-8924-2314301fab23\" (UID: \"d9770306-e191-4954-8924-2314301fab23\") "
Oct 30 13:19:30.796790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.796760    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9770306-e191-4954-8924-2314301fab23-kube-api-access-4nwtr" pod "d9770306-e191-4954-8924-2314301fab23" (UID: "d9770306-e191-4954-8924-2314301fab23"). InnerVolumeSpecName "kube-api-access-4nwtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:30.896240 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:30.896192    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nwtr\" (UniqueName: \"kubernetes.io/projected/d9770306-e191-4954-8924-2314301fab23-kube-api-access-4nwtr\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:19:31.092432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:31.091935    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"]
Oct 30 13:19:31.097155 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:19:31.097116    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d0f798_31cc_4548_999c_40c60f7ff934.slice/crio-ac0b3220c326b1812ba5dd033d2af6162402c335d58e23cb29cd142b40df6ae3 WatchSource:0}: Error finding container ac0b3220c326b1812ba5dd033d2af6162402c335d58e23cb29cd142b40df6ae3: Status 404 returned error can't find the container with id ac0b3220c326b1812ba5dd033d2af6162402c335d58e23cb29cd142b40df6ae3
Oct 30 13:19:31.433965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:31.433930    4429 generic.go:356] "Generic (PLEG): container finished" podID="d9770306-e191-4954-8924-2314301fab23" containerID="baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d" exitCode=137
Oct 30 13:19:31.434134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:31.433982    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:31.434134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:31.434048    4429 scope.go:122] "RemoveContainer" containerID="baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d"
Oct 30 13:19:31.435859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:31.435804    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9" event={"ID":"06d0f798-31cc-4548-999c-40c60f7ff934","Type":"ContainerStarted","Data":"ac0b3220c326b1812ba5dd033d2af6162402c335d58e23cb29cd142b40df6ae3"}
Oct 30 13:19:31.448207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:31.448186    4429 scope.go:122] "RemoveContainer" containerID="baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d"
Oct 30 13:19:31.448619 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:31.448597    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d\": container with ID starting with baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d not found: ID does not exist" containerID="baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d"
Oct 30 13:19:31.448692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:31.448636    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d"} err="failed to get container status \"baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d\": rpc error: code = NotFound desc = could not find container \"baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d\": container with ID starting with baabe86f3a245128a438793e8e59c44b7b6df68b5263950033e3c0fe3b24ec6d not found: ID does not exist"
Oct 30 13:19:31.499786 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:31.499709    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-8333e1\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="d9770306-e191-4954-8924-2314301fab23" pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:32.076177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:32.076141    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9770306-e191-4954-8924-2314301fab23" path="/var/lib/kubelet/pods/d9770306-e191-4954-8924-2314301fab23/volumes"
Oct 30 13:19:32.086611 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:32.086558    4429 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11532-8333e1\" is forbidden: User \"system:node:ip-10-0-136-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-246.ec2.internal' and this object" podUID="d9770306-e191-4954-8924-2314301fab23" pod="opendatahub/test-curl-11532-8333e1"
Oct 30 13:19:32.444966 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:32.444930    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9" event={"ID":"06d0f798-31cc-4548-999c-40c60f7ff934","Type":"ContainerStarted","Data":"6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec"}
Oct 30 13:19:32.445184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:32.445030    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:32.500308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:32.500225    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9" podStartSLOduration=3.175251705 podStartE2EDuration="3.500206432s" totalImagesPullingTime="324.954727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:32.496752647 +0000 UTC m=+2830.552246471" watchObservedRunningTime="2026-10-30 13:19:32.500206432 +0000 UTC m=+2830.555700255"
Oct 30 13:19:35.466683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:35.466599    4429 generic.go:356] "Generic (PLEG): container finished" podID="d920225c-2d94-4464-98af-fa12dc2428fd" containerID="d85949087494873ab4cd8bacb282aedb466668f17aa5e1368437d8f926f9e800" exitCode=0
Oct 30 13:19:35.466683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:35.466670    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" event={"ID":"d920225c-2d94-4464-98af-fa12dc2428fd","Type":"ContainerDied","Data":"d85949087494873ab4cd8bacb282aedb466668f17aa5e1368437d8f926f9e800"}
Oct 30 13:19:35.467193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:35.467098    4429 scope.go:122] "RemoveContainer" containerID="d85949087494873ab4cd8bacb282aedb466668f17aa5e1368437d8f926f9e800"
Oct 30 13:19:35.468869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:35.468841    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:19:35.468973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:35.468909    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:19:35.468973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:35.468960    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:19:36.490154 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:36.490120    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" event={"ID":"d920225c-2d94-4464-98af-fa12dc2428fd","Type":"ContainerStarted","Data":"c82d3a744ab0596d8eb2cb4abf8e07060eeaa24107c7895bd48c838647dd09c7"}
Oct 30 13:19:36.619316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:36.619248    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:19:36.619517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:36.619429    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:19:36.619517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:36.619478    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:19:37.619411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:37.619376    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:19:37.619813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:37.619453    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:19:38.473838 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:38.473807    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:19:38.619910 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:38.619880    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:19:38.620307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:38.619944    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:19:39.619184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:39.619143    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:19:39.619439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:39.619226    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:19:40.619690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:40.619650    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:19:40.620142 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:40.619711    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:19:41.620057 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:41.620024    4429 patch_prober.go:29] interesting pod/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:19:41.620660 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:41.620087    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:19:41.925679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:41.925587    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:19:42.373567 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:19:42.373533    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:19:42.624915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:42.624844    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:19:42.625966 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:42.625946    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:19:59.123207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.123170    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"]
Oct 30 13:19:59.127566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.127547    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:19:59.148482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.148449    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"]
Oct 30 13:19:59.157748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.157720    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-4db150-dockercfg-ftbjb\""
Oct 30 13:19:59.157748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.157720    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-4db150\""
Oct 30 13:19:59.209552 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.209503    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"]
Oct 30 13:19:59.258624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.258592    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"]
Oct 30 13:19:59.258927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.258726    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:19:59.264549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.264513    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88kb\" (UniqueName: \"kubernetes.io/projected/b0d701f4-05ec-48c8-898b-d2b954e8f368-kube-api-access-c88kb\") pod \"payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75\" (UID: \"b0d701f4-05ec-48c8-898b-d2b954e8f368\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:19:59.264868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.264821    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d701f4-05ec-48c8-898b-d2b954e8f368-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75\" (UID: \"b0d701f4-05ec-48c8-898b-d2b954e8f368\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:19:59.366530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.366485    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88kb\" (UniqueName: \"kubernetes.io/projected/b0d701f4-05ec-48c8-898b-d2b954e8f368-kube-api-access-c88kb\") pod \"payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75\" (UID: \"b0d701f4-05ec-48c8-898b-d2b954e8f368\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:19:59.366530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.366529    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d701f4-05ec-48c8-898b-d2b954e8f368-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75\" (UID: \"b0d701f4-05ec-48c8-898b-d2b954e8f368\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:19:59.366801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.366774    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8nj\" (UniqueName: \"kubernetes.io/projected/d75912f6-090d-4853-8c85-c714af76d1f4-kube-api-access-rs8nj\") pod \"payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5\" (UID: \"d75912f6-090d-4853-8c85-c714af76d1f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:19:59.366868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.366834    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75912f6-090d-4853-8c85-c714af76d1f4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5\" (UID: \"d75912f6-090d-4853-8c85-c714af76d1f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:19:59.367380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.367349    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d701f4-05ec-48c8-898b-d2b954e8f368-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75\" (UID: \"b0d701f4-05ec-48c8-898b-d2b954e8f368\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:19:59.396191 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.396119    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88kb\" (UniqueName: \"kubernetes.io/projected/b0d701f4-05ec-48c8-898b-d2b954e8f368-kube-api-access-c88kb\") pod \"payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75\" (UID: \"b0d701f4-05ec-48c8-898b-d2b954e8f368\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:19:59.442550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.442514    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:19:59.468926 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.468880    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8nj\" (UniqueName: \"kubernetes.io/projected/d75912f6-090d-4853-8c85-c714af76d1f4-kube-api-access-rs8nj\") pod \"payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5\" (UID: \"d75912f6-090d-4853-8c85-c714af76d1f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:19:59.469107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.468967    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75912f6-090d-4853-8c85-c714af76d1f4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5\" (UID: \"d75912f6-090d-4853-8c85-c714af76d1f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:19:59.469778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.469752    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75912f6-090d-4853-8c85-c714af76d1f4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5\" (UID: \"d75912f6-090d-4853-8c85-c714af76d1f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:19:59.494542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.494514    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8nj\" (UniqueName: \"kubernetes.io/projected/d75912f6-090d-4853-8c85-c714af76d1f4-kube-api-access-rs8nj\") pod \"payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5\" (UID: \"d75912f6-090d-4853-8c85-c714af76d1f4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:19:59.593088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.593048    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:19:59.754873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.754836    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"]
Oct 30 13:19:59.756692 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:19:59.756663    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d701f4_05ec_48c8_898b_d2b954e8f368.slice/crio-e21cfbe7e724f67d8f84a69773a5a6ca92535c1e6f3d956658bea89da61c4747 WatchSource:0}: Error finding container e21cfbe7e724f67d8f84a69773a5a6ca92535c1e6f3d956658bea89da61c4747: Status 404 returned error can't find the container with id e21cfbe7e724f67d8f84a69773a5a6ca92535c1e6f3d956658bea89da61c4747
Oct 30 13:19:59.903486 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:19:59.903456    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75912f6_090d_4853_8c85_c714af76d1f4.slice/crio-6dc7eeb9bd37b22fb71b7fd9e7aaec0dbb29908e13d91254c2e8e62b575d592c WatchSource:0}: Error finding container 6dc7eeb9bd37b22fb71b7fd9e7aaec0dbb29908e13d91254c2e8e62b575d592c: Status 404 returned error can't find the container with id 6dc7eeb9bd37b22fb71b7fd9e7aaec0dbb29908e13d91254c2e8e62b575d592c
Oct 30 13:19:59.903833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:19:59.903810    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"]
Oct 30 13:20:00.672506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:00.672467    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5" event={"ID":"d75912f6-090d-4853-8c85-c714af76d1f4","Type":"ContainerStarted","Data":"806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2"}
Oct 30 13:20:00.673016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:00.672527    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5" event={"ID":"d75912f6-090d-4853-8c85-c714af76d1f4","Type":"ContainerStarted","Data":"6dc7eeb9bd37b22fb71b7fd9e7aaec0dbb29908e13d91254c2e8e62b575d592c"}
Oct 30 13:20:00.673016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:00.672556    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:20:00.674661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:00.674630    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75" event={"ID":"b0d701f4-05ec-48c8-898b-d2b954e8f368","Type":"ContainerStarted","Data":"46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c"}
Oct 30 13:20:00.674804 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:00.674681    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75" event={"ID":"b0d701f4-05ec-48c8-898b-d2b954e8f368","Type":"ContainerStarted","Data":"e21cfbe7e724f67d8f84a69773a5a6ca92535c1e6f3d956658bea89da61c4747"}
Oct 30 13:20:00.674804 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:00.674708    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:20:00.717959 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:00.717901    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5" podStartSLOduration=1.7178874560000001 podStartE2EDuration="1.717887456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:00.717460502 +0000 UTC m=+2858.772954329" watchObservedRunningTime="2026-10-30 13:20:00.717887456 +0000 UTC m=+2858.773381281"
Oct 30 13:20:00.764523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:00.764462    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75" podStartSLOduration=1.764448502 podStartE2EDuration="1.764448502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:00.762699027 +0000 UTC m=+2858.818192870" watchObservedRunningTime="2026-10-30 13:20:00.764448502 +0000 UTC m=+2858.819942349"
Oct 30 13:20:06.879323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:06.879273    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"]
Oct 30 13:20:06.884948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:06.884923    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:06.932309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:06.922679    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-63499e-serving-cert\""
Oct 30 13:20:06.940044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:06.940016    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"]
Oct 30 13:20:06.980927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:06.980898    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxkt\" (UniqueName: \"kubernetes.io/projected/eb361f7e-e214-47b7-bc86-e37708877f24-kube-api-access-2bxkt\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:06.981089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:06.981063    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-maas-api-tls\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:06.981160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:06.981140    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb361f7e-e214-47b7-bc86-e37708877f24-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:06.981198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:06.981188    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-metrics-certs\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.082258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.082217    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-metrics-certs\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.082483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.082351    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxkt\" (UniqueName: \"kubernetes.io/projected/eb361f7e-e214-47b7-bc86-e37708877f24-kube-api-access-2bxkt\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.082571 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.082524    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-maas-api-tls\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.082632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.082586    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb361f7e-e214-47b7-bc86-e37708877f24-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.083422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.083400    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb361f7e-e214-47b7-bc86-e37708877f24-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.085050 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.085029    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-metrics-certs\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.085185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.085165    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-maas-api-tls\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.108013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.107986    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxkt\" (UniqueName: \"kubernetes.io/projected/eb361f7e-e214-47b7-bc86-e37708877f24-kube-api-access-2bxkt\") pod \"maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.200672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.200641    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:07.506456 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:20:07.506423    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb361f7e_e214_47b7_bc86_e37708877f24.slice/crio-df29d89e83144be5f0323eb19a1f1c365c78276a722ec674905005b031de2ff1 WatchSource:0}: Error finding container df29d89e83144be5f0323eb19a1f1c365c78276a722ec674905005b031de2ff1: Status 404 returned error can't find the container with id df29d89e83144be5f0323eb19a1f1c365c78276a722ec674905005b031de2ff1
Oct 30 13:20:07.507547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.507525    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"]
Oct 30 13:20:07.745606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:07.745572    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9" event={"ID":"eb361f7e-e214-47b7-bc86-e37708877f24","Type":"ContainerStarted","Data":"df29d89e83144be5f0323eb19a1f1c365c78276a722ec674905005b031de2ff1"}
Oct 30 13:20:08.753934 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:08.753901    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9" event={"ID":"eb361f7e-e214-47b7-bc86-e37708877f24","Type":"ContainerStarted","Data":"1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13"}
Oct 30 13:20:08.754353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:08.753971    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:08.802958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:08.802897    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9" podStartSLOduration=2.406138472 podStartE2EDuration="2.802881678s" totalImagesPullingTime="396.743206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:08.79932154 +0000 UTC m=+2866.854815365" watchObservedRunningTime="2026-10-30 13:20:08.802881678 +0000 UTC m=+2866.858375502"
Oct 30 13:20:11.685177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:11.685149    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:20:11.685703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:11.685222    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:20:12.373247 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:20:12.373210    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:20:14.766638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:14.766611    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:20:22.235567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:22.235538    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"]
Oct 30 13:20:27.236495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.236444    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv" podUID="bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3" containerName="payload-processing" containerID="cri-o://2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa" gracePeriod=25
Oct 30 13:20:27.507214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.507146    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:20:27.661386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.661359    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-plugins-config-volume\") pod \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\" (UID: \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\") "
Oct 30 13:20:27.661555 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.661393    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-kube-api-access-67f2j\" (UniqueName: \"kubernetes.io/projected/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-kube-api-access-67f2j\") pod \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\" (UID: \"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3\") "
Oct 30 13:20:27.661737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.661716    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-plugins-config-volume" pod "bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3" (UID: "bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:27.661912 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.661899    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:20:27.663498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.663478    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-kube-api-access-67f2j" pod "bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3" (UID: "bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3"). InnerVolumeSpecName "kube-api-access-67f2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:27.763756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.763661    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67f2j\" (UniqueName: \"kubernetes.io/projected/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3-kube-api-access-67f2j\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:20:27.901652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.901617    4429 generic.go:356] "Generic (PLEG): container finished" podID="bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3" containerID="2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa" exitCode=0
Oct 30 13:20:27.901837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.901709    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"
Oct 30 13:20:27.901837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.901708    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv" event={"ID":"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3","Type":"ContainerDied","Data":"2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa"}
Oct 30 13:20:27.901837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.901761    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv" event={"ID":"bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3","Type":"ContainerDied","Data":"102ae9d3f16a730f6c66dcd46863e398c1a3d0d4a0cb29d712ca63df8af9c035"}
Oct 30 13:20:27.901837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.901783    4429 scope.go:122] "RemoveContainer" containerID="2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa"
Oct 30 13:20:27.912037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.912019    4429 scope.go:122] "RemoveContainer" containerID="2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa"
Oct 30 13:20:27.912362 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:20:27.912341    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa\": container with ID starting with 2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa not found: ID does not exist" containerID="2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa"
Oct 30 13:20:27.912460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.912370    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa"} err="failed to get container status \"2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa\": rpc error: code = NotFound desc = could not find container \"2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa\": container with ID starting with 2907309320d226b2d2d63501de6a690e8c59350a1a0a3f9747a5bbb597c18cfa not found: ID does not exist"
Oct 30 13:20:27.959978 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.959948    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"]
Oct 30 13:20:27.973454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:27.973425    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-182377-5769fc56b6-h4sdv"]
Oct 30 13:20:28.077001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:28.076932    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3" path="/var/lib/kubelet/pods/bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3/volumes"
Oct 30 13:20:39.978119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:39.978086    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"]
Oct 30 13:20:39.978996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:39.978976    4429 cpu_manager.go:414] "containerMap: removing container" podUID="bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3" containerName="payload-processing"
Oct 30 13:20:39.979063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:39.979001    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3" containerName="payload-processing"
Oct 30 13:20:39.979131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:39.979121    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf09fc6-0e8b-43fa-a2af-70a395bdf0a3" containerName="payload-processing"
Oct 30 13:20:39.984831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:39.984811    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:40.015763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.015737    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"]
Oct 30 13:20:40.025505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.025484    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2d2a0b-dockercfg-6d87c\""
Oct 30 13:20:40.035371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.035336    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3da904d-79b0-4f10-b122-fb0720de4008-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb\" (UID: \"b3da904d-79b0-4f10-b122-fb0720de4008\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:40.035504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.035385    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47k7\" (UniqueName: \"kubernetes.io/projected/b3da904d-79b0-4f10-b122-fb0720de4008-kube-api-access-n47k7\") pod \"payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb\" (UID: \"b3da904d-79b0-4f10-b122-fb0720de4008\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:40.107772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.106627    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2d2a0b\""
Oct 30 13:20:40.137660 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.137617    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3da904d-79b0-4f10-b122-fb0720de4008-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb\" (UID: \"b3da904d-79b0-4f10-b122-fb0720de4008\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:40.137866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.137685    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47k7\" (UniqueName: \"kubernetes.io/projected/b3da904d-79b0-4f10-b122-fb0720de4008-kube-api-access-n47k7\") pod \"payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb\" (UID: \"b3da904d-79b0-4f10-b122-fb0720de4008\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:40.139140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.139113    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3da904d-79b0-4f10-b122-fb0720de4008-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb\" (UID: \"b3da904d-79b0-4f10-b122-fb0720de4008\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:40.163717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.163678    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47k7\" (UniqueName: \"kubernetes.io/projected/b3da904d-79b0-4f10-b122-fb0720de4008-kube-api-access-n47k7\") pod \"payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb\" (UID: \"b3da904d-79b0-4f10-b122-fb0720de4008\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:40.305463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.305384    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:40.611738 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:20:40.611705    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3da904d_79b0_4f10_b122_fb0720de4008.slice/crio-77acaefff0d78a044bae8f9b7f42d186cce7f61ebd0cc25589bcb3d4486f9726 WatchSource:0}: Error finding container 77acaefff0d78a044bae8f9b7f42d186cce7f61ebd0cc25589bcb3d4486f9726: Status 404 returned error can't find the container with id 77acaefff0d78a044bae8f9b7f42d186cce7f61ebd0cc25589bcb3d4486f9726
Oct 30 13:20:40.612398 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:40.612374    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"]
Oct 30 13:20:41.003009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:41.002971    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb" event={"ID":"b3da904d-79b0-4f10-b122-fb0720de4008","Type":"ContainerStarted","Data":"5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2"}
Oct 30 13:20:41.003492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:41.003041    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb" event={"ID":"b3da904d-79b0-4f10-b122-fb0720de4008","Type":"ContainerStarted","Data":"77acaefff0d78a044bae8f9b7f42d186cce7f61ebd0cc25589bcb3d4486f9726"}
Oct 30 13:20:41.003492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:41.003204    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:20:41.049116 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:41.049064    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb" podStartSLOduration=2.049049031 podStartE2EDuration="2.049049031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:41.046505639 +0000 UTC m=+2899.101999466" watchObservedRunningTime="2026-10-30 13:20:41.049049031 +0000 UTC m=+2899.104542856"
Oct 30 13:20:42.373951 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:20:42.373562    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:20:52.012939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:20:52.012909    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:21:12.373633 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:21:12.373544    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:21:21.885875 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:21.885839    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"]
Oct 30 13:21:21.886608 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:21.886086    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9" podUID="06d0f798-31cc-4548-999c-40c60f7ff934" containerName="maas-api" containerID="cri-o://6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec" gracePeriod=30
Oct 30 13:21:22.099182 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.098738    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"]
Oct 30 13:21:22.106876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.104468    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:22.106876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.105882    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"]
Oct 30 13:21:22.122777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.122747    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-75d385-dockercfg-kbc9x\""
Oct 30 13:21:22.122777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.122747    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-75d385\""
Oct 30 13:21:22.178146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.178117    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e361b56b-e61d-4c0f-a915-da0415767b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5\" (UID: \"e361b56b-e61d-4c0f-a915-da0415767b13\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:22.178323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.178191    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92mz\" (UniqueName: \"kubernetes.io/projected/e361b56b-e61d-4c0f-a915-da0415767b13-kube-api-access-j92mz\") pod \"payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5\" (UID: \"e361b56b-e61d-4c0f-a915-da0415767b13\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:22.217519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.217483    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:21:22.279208 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.279167    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls\") pod \"06d0f798-31cc-4548-999c-40c60f7ff934\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") "
Oct 30 13:21:22.279881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.279845    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d0f798-31cc-4548-999c-40c60f7ff934-kube-api-access-rrhbf\" (UniqueName: \"kubernetes.io/projected/06d0f798-31cc-4548-999c-40c60f7ff934-kube-api-access-rrhbf\") pod \"06d0f798-31cc-4548-999c-40c60f7ff934\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") "
Oct 30 13:21:22.280098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.280082    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d0f798-31cc-4548-999c-40c60f7ff934-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d0f798-31cc-4548-999c-40c60f7ff934-openshift-service-ca\") pod \"06d0f798-31cc-4548-999c-40c60f7ff934\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") "
Oct 30 13:21:22.280558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.280510    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-metrics-certs\") pod \"06d0f798-31cc-4548-999c-40c60f7ff934\" (UID: \"06d0f798-31cc-4548-999c-40c60f7ff934\") "
Oct 30 13:21:22.281719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.281691    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92mz\" (UniqueName: \"kubernetes.io/projected/e361b56b-e61d-4c0f-a915-da0415767b13-kube-api-access-j92mz\") pod \"payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5\" (UID: \"e361b56b-e61d-4c0f-a915-da0415767b13\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:22.282190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.282166    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e361b56b-e61d-4c0f-a915-da0415767b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5\" (UID: \"e361b56b-e61d-4c0f-a915-da0415767b13\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:22.283377 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.283317    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e361b56b-e61d-4c0f-a915-da0415767b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5\" (UID: \"e361b56b-e61d-4c0f-a915-da0415767b13\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:22.288618 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.288592    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d0f798-31cc-4548-999c-40c60f7ff934-openshift-service-ca" pod "06d0f798-31cc-4548-999c-40c60f7ff934" (UID: "06d0f798-31cc-4548-999c-40c60f7ff934"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:22.312412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.312350    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d0f798-31cc-4548-999c-40c60f7ff934-kube-api-access-rrhbf" pod "06d0f798-31cc-4548-999c-40c60f7ff934" (UID: "06d0f798-31cc-4548-999c-40c60f7ff934"). InnerVolumeSpecName "kube-api-access-rrhbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:22.312832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.312811    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls" pod "06d0f798-31cc-4548-999c-40c60f7ff934" (UID: "06d0f798-31cc-4548-999c-40c60f7ff934"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:22.313018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.313002    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-metrics-certs" pod "06d0f798-31cc-4548-999c-40c60f7ff934" (UID: "06d0f798-31cc-4548-999c-40c60f7ff934"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:22.319326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.317156    4429 generic.go:356] "Generic (PLEG): container finished" podID="06d0f798-31cc-4548-999c-40c60f7ff934" containerID="6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec" exitCode=0
Oct 30 13:21:22.319326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.317441    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9" event={"ID":"06d0f798-31cc-4548-999c-40c60f7ff934","Type":"ContainerDied","Data":"6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec"}
Oct 30 13:21:22.319326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.317496    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9" event={"ID":"06d0f798-31cc-4548-999c-40c60f7ff934","Type":"ContainerDied","Data":"ac0b3220c326b1812ba5dd033d2af6162402c335d58e23cb29cd142b40df6ae3"}
Oct 30 13:21:22.319326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.317527    4429 scope.go:122] "RemoveContainer" containerID="6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec"
Oct 30 13:21:22.319326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.317693    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"
Oct 30 13:21:22.329877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.329850    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92mz\" (UniqueName: \"kubernetes.io/projected/e361b56b-e61d-4c0f-a915-da0415767b13-kube-api-access-j92mz\") pod \"payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5\" (UID: \"e361b56b-e61d-4c0f-a915-da0415767b13\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:22.353328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.353305    4429 scope.go:122] "RemoveContainer" containerID="6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec"
Oct 30 13:21:22.353713 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:21:22.353688    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec\": container with ID starting with 6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec not found: ID does not exist" containerID="6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec"
Oct 30 13:21:22.353806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.353727    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec"} err="failed to get container status \"6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec\": rpc error: code = NotFound desc = could not find container \"6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec\": container with ID starting with 6ad4576a53d803e584b99accbcab50b84084fa7b56d378fb6e176e3b645453ec not found: ID does not exist"
Oct 30 13:21:22.384197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.384163    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:21:22.384197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.384195    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06d0f798-31cc-4548-999c-40c60f7ff934-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:21:22.384197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.384205    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrhbf\" (UniqueName: \"kubernetes.io/projected/06d0f798-31cc-4548-999c-40c60f7ff934-kube-api-access-rrhbf\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:21:22.384197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.384214    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d0f798-31cc-4548-999c-40c60f7ff934-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:21:22.397402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.397369    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"]
Oct 30 13:21:22.403711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.403682    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-037a5a-6758fbd48-fbgd9"]
Oct 30 13:21:22.439996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.439963    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:22.876753 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:21:22.876717    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode361b56b_e61d_4c0f_a915_da0415767b13.slice/crio-cb17be68249630ef95b4495849af32067a298c188453f2cd01b6459448a2297d WatchSource:0}: Error finding container cb17be68249630ef95b4495849af32067a298c188453f2cd01b6459448a2297d: Status 404 returned error can't find the container with id cb17be68249630ef95b4495849af32067a298c188453f2cd01b6459448a2297d
Oct 30 13:21:22.877253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:22.877234    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"]
Oct 30 13:21:23.325888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:23.325852    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5" event={"ID":"e361b56b-e61d-4c0f-a915-da0415767b13","Type":"ContainerStarted","Data":"a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9"}
Oct 30 13:21:23.326393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:23.325913    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5" event={"ID":"e361b56b-e61d-4c0f-a915-da0415767b13","Type":"ContainerStarted","Data":"cb17be68249630ef95b4495849af32067a298c188453f2cd01b6459448a2297d"}
Oct 30 13:21:23.326393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:23.325944    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:23.380624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:23.380570    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5" podStartSLOduration=1.380552333 podStartE2EDuration="1.380552333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:23.376752371 +0000 UTC m=+2941.432246195" watchObservedRunningTime="2026-10-30 13:21:23.380552333 +0000 UTC m=+2941.436046158"
Oct 30 13:21:23.525807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:23.525777    4429 scope.go:122] "RemoveContainer" containerID="fb3a4783813170092a42730a22c14efd83b193b728e3e4de7508571361a72756"
Oct 30 13:21:24.076645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:24.076608    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d0f798-31cc-4548-999c-40c60f7ff934" path="/var/lib/kubelet/pods/06d0f798-31cc-4548-999c-40c60f7ff934/volumes"
Oct 30 13:21:33.405369 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.405334    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"]
Oct 30 13:21:33.406256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.406222    4429 cpu_manager.go:414] "containerMap: removing container" podUID="06d0f798-31cc-4548-999c-40c60f7ff934" containerName="maas-api"
Oct 30 13:21:33.406256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.406247    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d0f798-31cc-4548-999c-40c60f7ff934" containerName="maas-api"
Oct 30 13:21:33.406506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.406457    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d0f798-31cc-4548-999c-40c60f7ff934" containerName="maas-api"
Oct 30 13:21:33.424296 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.424246    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.449184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.449154    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"]
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.450492    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f924a4-fb89-4052-bf95-790c1129da55-istio-podinfo\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.450555    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-socket\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.450645    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-data\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.450744    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-envoy\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.450772    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-certs\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.450853    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j98mk\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-kube-api-access-j98mk\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.450931    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-credential-socket\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.450955    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f924a4-fb89-4052-bf95-790c1129da55-istiod-ca-cert\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.451056    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-istio-token\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.456091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.452510    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0505b0-openshift-default-dockercfg-chvsx\""
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562159    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-certs\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562229    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j98mk\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-kube-api-access-j98mk\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562343    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-credential-socket\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562383    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f924a4-fb89-4052-bf95-790c1129da55-istiod-ca-cert\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562425    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-istio-token\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562544    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f924a4-fb89-4052-bf95-790c1129da55-istio-podinfo\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562601    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-socket\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562634    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-data\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.568357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.562750    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-envoy\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.607942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.573029    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f924a4-fb89-4052-bf95-790c1129da55-istiod-ca-cert\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.607942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.573259    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-certs\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.607942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.589004    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-credential-socket\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.607942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.589905    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-data\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.607942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.590763    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-socket\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.607942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.592155    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-envoy\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.607942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.604714    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f924a4-fb89-4052-bf95-790c1129da55-istio-podinfo\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.619843 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.619804    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-istio-token\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.621068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.621043    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j98mk\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-kube-api-access-j98mk\") pod \"e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") " pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.745980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.745944    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:33.786518 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.786489    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"]
Oct 30 13:21:33.793232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.793201    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:33.847150 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.847117    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"]
Oct 30 13:21:33.863353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.857531    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-8740d4-serving-cert\""
Oct 30 13:21:33.968277 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.968237    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm92n\" (UniqueName: \"kubernetes.io/projected/9017cc61-e4f0-403a-90f9-a0bebe754590-kube-api-access-lm92n\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:33.968478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.968447    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:33.968478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.968471    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-metrics-certs\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:33.968597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:33.968509    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017cc61-e4f0-403a-90f9-a0bebe754590-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.069551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.069411    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm92n\" (UniqueName: \"kubernetes.io/projected/9017cc61-e4f0-403a-90f9-a0bebe754590-kube-api-access-lm92n\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.069551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.069512    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.069551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.069543    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-metrics-certs\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.069755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.069590    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017cc61-e4f0-403a-90f9-a0bebe754590-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.070355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.070327    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017cc61-e4f0-403a-90f9-a0bebe754590-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.072057 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:21:34.072022    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f924a4_fb89_4052_bf95_790c1129da55.slice/crio-aefed12d19ae42e53879c884b532d53a49ee0617205f2f7b13d13ca79335d327 WatchSource:0}: Error finding container aefed12d19ae42e53879c884b532d53a49ee0617205f2f7b13d13ca79335d327: Status 404 returned error can't find the container with id aefed12d19ae42e53879c884b532d53a49ee0617205f2f7b13d13ca79335d327
Oct 30 13:21:34.072857 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.072815    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-metrics-certs\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.072857 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.072843    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.074901 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.074868    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:34.075002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.074941    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:34.075002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.074976    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:34.078022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.078002    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"]
Oct 30 13:21:34.095447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.095415    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm92n\" (UniqueName: \"kubernetes.io/projected/9017cc61-e4f0-403a-90f9-a0bebe754590-kube-api-access-lm92n\") pod \"maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.111205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.111180    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:34.337656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.337559    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:21:34.415156 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.414430    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" event={"ID":"d3f924a4-fb89-4052-bf95-790c1129da55","Type":"ContainerStarted","Data":"8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560"}
Oct 30 13:21:34.415156 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.414519    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" event={"ID":"d3f924a4-fb89-4052-bf95-790c1129da55","Type":"ContainerStarted","Data":"aefed12d19ae42e53879c884b532d53a49ee0617205f2f7b13d13ca79335d327"}
Oct 30 13:21:34.427671 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:21:34.427631    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9017cc61_e4f0_403a_90f9_a0bebe754590.slice/crio-a88aca376b4515fbe3fa97c3fdc851e7735302e744215e4d8f59606a3886e94d WatchSource:0}: Error finding container a88aca376b4515fbe3fa97c3fdc851e7735302e744215e4d8f59606a3886e94d: Status 404 returned error can't find the container with id a88aca376b4515fbe3fa97c3fdc851e7735302e744215e4d8f59606a3886e94d
Oct 30 13:21:34.429928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.429874    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"]
Oct 30 13:21:34.472941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.472870    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" podStartSLOduration=1.472853689 podStartE2EDuration="1.472853689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:34.469931511 +0000 UTC m=+2952.525425334" watchObservedRunningTime="2026-10-30 13:21:34.472853689 +0000 UTC m=+2952.528347512"
Oct 30 13:21:34.746470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:34.746398    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:35.423363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:35.423325    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw" event={"ID":"9017cc61-e4f0-403a-90f9-a0bebe754590","Type":"ContainerStarted","Data":"0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d"}
Oct 30 13:21:35.423856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:35.423391    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw" event={"ID":"9017cc61-e4f0-403a-90f9-a0bebe754590","Type":"ContainerStarted","Data":"a88aca376b4515fbe3fa97c3fdc851e7735302e744215e4d8f59606a3886e94d"}
Oct 30 13:21:35.423856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:35.423621    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:35.471476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:35.471427    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw" podStartSLOduration=2.080309699 podStartE2EDuration="2.471412642s" totalImagesPullingTime="391.102943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:35.468663181 +0000 UTC m=+2953.524157017" watchObservedRunningTime="2026-10-30 13:21:35.471412642 +0000 UTC m=+2953.526906654"
Oct 30 13:21:35.752777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:35.752748    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:36.430844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:36.430813    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:36.431744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:36.431725    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:21:41.436598 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:41.436570    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:21:42.373152 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:21:42.373108    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:21:49.355957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.355927    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"]
Oct 30 13:21:49.360584 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.360564    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.384323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.383625    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"]
Oct 30 13:21:49.394424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.391943    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-acf9e2-serving-cert\""
Oct 30 13:21:49.517660 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.517626    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-metrics-certs\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.517837 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.517675    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.517921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.517838    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq9qc\" (UniqueName: \"kubernetes.io/projected/88228612-d786-4f85-9cea-076c8db124a4-kube-api-access-pq9qc\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.517921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.517889    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88228612-d786-4f85-9cea-076c8db124a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.619038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.618954    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.619188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.619079    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq9qc\" (UniqueName: \"kubernetes.io/projected/88228612-d786-4f85-9cea-076c8db124a4-kube-api-access-pq9qc\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.619188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.619119    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88228612-d786-4f85-9cea-076c8db124a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.619316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.619212    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-metrics-certs\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.619854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.619829    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88228612-d786-4f85-9cea-076c8db124a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.621749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.621732    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.621813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.621794    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-metrics-certs\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.644266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.644242    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq9qc\" (UniqueName: \"kubernetes.io/projected/88228612-d786-4f85-9cea-076c8db124a4-kube-api-access-pq9qc\") pod \"maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.676965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.676940    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:49.980599 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:21:49.980570    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88228612_d786_4f85_9cea_076c8db124a4.slice/crio-f241c44d0055ac2e097242e628f62d86ae99d971b20f28e9ee17286608b11a1f WatchSource:0}: Error finding container f241c44d0055ac2e097242e628f62d86ae99d971b20f28e9ee17286608b11a1f: Status 404 returned error can't find the container with id f241c44d0055ac2e097242e628f62d86ae99d971b20f28e9ee17286608b11a1f
Oct 30 13:21:49.981014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:49.980995    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"]
Oct 30 13:21:50.534576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:50.534484    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m" event={"ID":"88228612-d786-4f85-9cea-076c8db124a4","Type":"ContainerStarted","Data":"583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae"}
Oct 30 13:21:50.536867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:50.536839    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m" event={"ID":"88228612-d786-4f85-9cea-076c8db124a4","Type":"ContainerStarted","Data":"f241c44d0055ac2e097242e628f62d86ae99d971b20f28e9ee17286608b11a1f"}
Oct 30 13:21:50.537115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:50.536929    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:21:50.588899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:50.588841    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m" podStartSLOduration=1.299554014 podStartE2EDuration="1.588821533s" totalImagesPullingTime="289.267519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:50.583348739 +0000 UTC m=+2968.638842569" watchObservedRunningTime="2026-10-30 13:21:50.588821533 +0000 UTC m=+2968.644315361"
Oct 30 13:21:56.551932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:21:56.548308    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:22:12.382214 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:22:12.382173    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:22:15.613977 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.613945    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"]
Oct 30 13:22:15.619845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.619827    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.641076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.641047    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"]
Oct 30 13:22:15.648161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.648136    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0505b0-serving-cert\""
Oct 30 13:22:15.783985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.783955    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6244f0-a709-46d6-bafd-dbad9edb2d74-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.784168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.784016    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rv7d\" (UniqueName: \"kubernetes.io/projected/4a6244f0-a709-46d6-bafd-dbad9edb2d74-kube-api-access-4rv7d\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.784168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.784118    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.784304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.784276    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-metrics-certs\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.886024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.885935    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6244f0-a709-46d6-bafd-dbad9edb2d74-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.886024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.886010    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rv7d\" (UniqueName: \"kubernetes.io/projected/4a6244f0-a709-46d6-bafd-dbad9edb2d74-kube-api-access-4rv7d\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.886217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.886050    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.886217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.886155    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-metrics-certs\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.886755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.886727    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6244f0-a709-46d6-bafd-dbad9edb2d74-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.888891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.888860    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-metrics-certs\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.889005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.888863    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.926085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.926061    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rv7d\" (UniqueName: \"kubernetes.io/projected/4a6244f0-a709-46d6-bafd-dbad9edb2d74-kube-api-access-4rv7d\") pod \"maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:15.935075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:15.935053    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:16.239675 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:22:16.239639    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6244f0_a709_46d6_bafd_dbad9edb2d74.slice/crio-3f589bbbd151d8572058f137f4ea36b388b52fbc4f3d2e66b4eac12a67784969 WatchSource:0}: Error finding container 3f589bbbd151d8572058f137f4ea36b388b52fbc4f3d2e66b4eac12a67784969: Status 404 returned error can't find the container with id 3f589bbbd151d8572058f137f4ea36b388b52fbc4f3d2e66b4eac12a67784969
Oct 30 13:22:16.240390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:16.240368    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"]
Oct 30 13:22:16.726877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:16.726836    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9" event={"ID":"4a6244f0-a709-46d6-bafd-dbad9edb2d74","Type":"ContainerStarted","Data":"f917b683a59a8ef507f75033dcbe2c521b0a9ba2d40e610e833768873e18694e"}
Oct 30 13:22:16.727329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:16.726900    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9" event={"ID":"4a6244f0-a709-46d6-bafd-dbad9edb2d74","Type":"ContainerStarted","Data":"3f589bbbd151d8572058f137f4ea36b388b52fbc4f3d2e66b4eac12a67784969"}
Oct 30 13:22:16.727329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:16.726931    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:16.795870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:16.795805    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9" podStartSLOduration=1.450566069 podStartE2EDuration="1.795785947s" totalImagesPullingTime="345.219878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:16.791494845 +0000 UTC m=+2994.846988697" watchObservedRunningTime="2026-10-30 13:22:16.795785947 +0000 UTC m=+2994.851279772"
Oct 30 13:22:22.741525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:22.741498    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:22:42.373355 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:22:42.373258    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:22:46.210755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.210716    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"]
Oct 30 13:22:46.216587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.216566    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:46.234516 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.234494    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"]
Oct 30 13:22:46.244454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.244434    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-88e9f3\""
Oct 30 13:22:46.336415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.336375    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w\" (UID: \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:46.336720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.336699    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lv65\" (UniqueName: \"kubernetes.io/projected/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-kube-api-access-4lv65\") pod \"payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w\" (UID: \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:46.438021 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.437986    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w\" (UID: \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:46.438201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.438124    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lv65\" (UniqueName: \"kubernetes.io/projected/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-kube-api-access-4lv65\") pod \"payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w\" (UID: \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:46.438682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.438663    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w\" (UID: \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:46.463480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.463414    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lv65\" (UniqueName: \"kubernetes.io/projected/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-kube-api-access-4lv65\") pod \"payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w\" (UID: \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:46.530937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.530903    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:46.841812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.841756    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"]
Oct 30 13:22:46.842057 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:22:46.842028    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f1e8a2_eb2a_4e11_8923_22a34e177c4a.slice/crio-53e0ae8d28e187343d52283c50e74564a848414d06f906e16f428de7c9d06a19 WatchSource:0}: Error finding container 53e0ae8d28e187343d52283c50e74564a848414d06f906e16f428de7c9d06a19: Status 404 returned error can't find the container with id 53e0ae8d28e187343d52283c50e74564a848414d06f906e16f428de7c9d06a19
Oct 30 13:22:46.968081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.968046    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w" event={"ID":"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a","Type":"ContainerStarted","Data":"5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf"}
Oct 30 13:22:46.968228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.968108    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w" event={"ID":"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a","Type":"ContainerStarted","Data":"53e0ae8d28e187343d52283c50e74564a848414d06f906e16f428de7c9d06a19"}
Oct 30 13:22:46.968228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:46.968136    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:22:47.014878 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:47.014828    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w" podStartSLOduration=1.014812831 podStartE2EDuration="1.014812831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:47.01201552 +0000 UTC m=+3025.067509362" watchObservedRunningTime="2026-10-30 13:22:47.014812831 +0000 UTC m=+3025.070306654"
Oct 30 13:22:51.485529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.485492    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"]
Oct 30 13:22:51.491456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.491436    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.508660 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.508637    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"]
Oct 30 13:22:51.510401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.510379    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"kube-root-ca.crt\""
Oct 30 13:22:51.518893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.518870    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"payload-processing-external-model-plugins-e2e-extmod-b-75d385\""
Oct 30 13:22:51.519044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.518936    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"openshift-service-ca.crt\""
Oct 30 13:22:51.524052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.524027    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"agc-cred-b982a13dfb55\""
Oct 30 13:22:51.524052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.524046    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"routing-overlay\""
Oct 30 13:22:51.524240 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.524191    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"payload-processing-external-model-e2e--e38313e4-dockercfg-wfbv7\""
Oct 30 13:22:51.610791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.610755    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qlz\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-kube-api-access-56qlz\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.610990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.610831    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.610990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.610867    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.611117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.610994    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.712505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.712474    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.712683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.712588    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.712756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.712742    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qlz\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-kube-api-access-56qlz\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.712820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.712803    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.713402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.713364    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.713562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.713521    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.715950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.715925    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.738435 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.738380    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qlz\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-kube-api-access-56qlz\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:51.805093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:51.805070    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:52.108715 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:22:52.108689    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a1dc4e_1f0e_4295_b2b3_f0193f7311f6.slice/crio-f1ddac16698a147ee789e5d6093633a0de8a20c9241e04933d0a051e56a60c02 WatchSource:0}: Error finding container f1ddac16698a147ee789e5d6093633a0de8a20c9241e04933d0a051e56a60c02: Status 404 returned error can't find the container with id f1ddac16698a147ee789e5d6093633a0de8a20c9241e04933d0a051e56a60c02
Oct 30 13:22:52.109599 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:52.109578    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"]
Oct 30 13:22:53.012518 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:53.012486    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg" event={"ID":"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6","Type":"ContainerStarted","Data":"7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7"}
Oct 30 13:22:53.012955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:53.012537    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg" event={"ID":"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6","Type":"ContainerStarted","Data":"f1ddac16698a147ee789e5d6093633a0de8a20c9241e04933d0a051e56a60c02"}
Oct 30 13:22:53.012955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:53.012703    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:22:53.062553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:53.062508    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg" podStartSLOduration=2.062496186 podStartE2EDuration="2.062496186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:53.061475062 +0000 UTC m=+3031.116968887" watchObservedRunningTime="2026-10-30 13:22:53.062496186 +0000 UTC m=+3031.117990011"
Oct 30 13:22:56.087569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:56.087483    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"]
Oct 30 13:22:56.088129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:56.087721    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy" containerID="cri-o://adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761" gracePeriod=30
Oct 30 13:22:57.977786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:22:57.977758    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:23:01.393075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.393048    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:23:01.473367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473334    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-certs\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.473548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473458    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-kube-api-access-ptxbv\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-kube-api-access-ptxbv\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.473619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473544    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-credential-socket\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.473619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473584    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-podinfo\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.473619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473613    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-data\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.473798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473660    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-socket\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.473798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473708    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istiod-ca-cert\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.473798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473741    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-token\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-token\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.473798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473774    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-envoy\") pod \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\" (UID: \"3a5e28f6-11d7-496c-acee-cc70e26a9be7\") "
Oct 30 13:23:01.475048 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.473702    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-certs" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:01.475220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.474423    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istiod-ca-cert" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:01.475404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.474648    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-data" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:01.475509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.474828    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-socket" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:01.475616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.475017    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-credential-socket" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:01.477797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.477761    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-podinfo" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:23:01.478382 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.478351    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-envoy" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:23:01.480104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.480077    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-token" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:01.480359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.480337    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-kube-api-access-ptxbv" pod "3a5e28f6-11d7-496c-acee-cc70e26a9be7" (UID: "3a5e28f6-11d7-496c-acee-cc70e26a9be7"). InnerVolumeSpecName "kube-api-access-ptxbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:01.576140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576105    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:01.576140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576133    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:01.576140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576143    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:01.576140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576151    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:01.576437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576161    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:01.576437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576169    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:01.576437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576176    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:01.576437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576184    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxbv\" (UniqueName: \"kubernetes.io/projected/3a5e28f6-11d7-496c-acee-cc70e26a9be7-kube-api-access-ptxbv\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:01.576437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:01.576192    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5e28f6-11d7-496c-acee-cc70e26a9be7-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:23:02.121374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.121339    4429 generic.go:356] "Generic (PLEG): container finished" podID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerID="adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761" exitCode=0
Oct 30 13:23:02.121533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.121382    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"
Oct 30 13:23:02.121533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.121400    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" event={"ID":"3a5e28f6-11d7-496c-acee-cc70e26a9be7","Type":"ContainerDied","Data":"adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761"}
Oct 30 13:23:02.121533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.121451    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9" event={"ID":"3a5e28f6-11d7-496c-acee-cc70e26a9be7","Type":"ContainerDied","Data":"70ced8f2c2547c7a89f6a7914ce0fa1647de76c5a64e539255e7d596c6a1f33a"}
Oct 30 13:23:02.121533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.121478    4429 scope.go:122] "RemoveContainer" containerID="adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761"
Oct 30 13:23:02.133107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.133091    4429 scope.go:122] "RemoveContainer" containerID="adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761"
Oct 30 13:23:02.133420 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:23:02.133396    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761\": container with ID starting with adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761 not found: ID does not exist" containerID="adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761"
Oct 30 13:23:02.133522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.133426    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761"} err="failed to get container status \"adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761\": rpc error: code = NotFound desc = could not find container \"adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761\": container with ID starting with adf83808959d26b0e37b968de52098ef885581522dc80c25d2ef9e5055159761 not found: ID does not exist"
Oct 30 13:23:02.195906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.195877    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"]
Oct 30 13:23:02.208639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:02.208615    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-pt9z9"]
Oct 30 13:23:04.024536 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:04.024483    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:23:04.078145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:04.078114    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" path="/var/lib/kubelet/pods/3a5e28f6-11d7-496c-acee-cc70e26a9be7/volumes"
Oct 30 13:23:05.145425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:05.145322    4429 generic.go:356] "Generic (PLEG): container finished" podID="d3f924a4-fb89-4052-bf95-790c1129da55" containerID="8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560" exitCode=0
Oct 30 13:23:05.145425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:05.145403    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" event={"ID":"d3f924a4-fb89-4052-bf95-790c1129da55","Type":"ContainerDied","Data":"8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560"}
Oct 30 13:23:05.145890 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:05.145811    4429 scope.go:122] "RemoveContainer" containerID="8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560"
Oct 30 13:23:05.147612 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:05.147588    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:23:05.147686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:05.147654    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:23:05.147729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:05.147687    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:23:06.180159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:06.180130    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" event={"ID":"d3f924a4-fb89-4052-bf95-790c1129da55","Type":"ContainerStarted","Data":"8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d"}
Oct 30 13:23:06.431589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:06.431511    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:23:06.746454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:06.746416    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:23:06.772447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:06.772409    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:23:06.772615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:06.772468    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 13:23:07.746236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:07.746202    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:23:07.746623 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:07.746258    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 13:23:08.756311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:08.747340    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:23:08.756311 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:08.747402    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 13:23:09.747002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:09.746970    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:23:09.747204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:09.747024    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 13:23:10.746652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:10.746615    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:23:10.747025 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:10.746676    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 13:23:11.446228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.446195    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"]
Oct 30 13:23:11.446959 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.446944    4429 cpu_manager.go:414] "containerMap: removing container" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy"
Oct 30 13:23:11.447005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.446962    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy"
Oct 30 13:23:11.447080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.447071    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5e28f6-11d7-496c-acee-cc70e26a9be7" containerName="istio-proxy"
Oct 30 13:23:11.450860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.450840    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:11.475335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.475308    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"]
Oct 30 13:23:11.539411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.539375    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5n9t\" (UniqueName: \"kubernetes.io/projected/7f992cba-9c34-45a5-abc1-bfa307a8badb-kube-api-access-r5n9t\") pod \"payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w\" (UID: \"7f992cba-9c34-45a5-abc1-bfa307a8badb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:11.539594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.539478    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f992cba-9c34-45a5-abc1-bfa307a8badb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w\" (UID: \"7f992cba-9c34-45a5-abc1-bfa307a8badb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:11.542076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.542057    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0d9115\""
Oct 30 13:23:11.645274 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.645218    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5n9t\" (UniqueName: \"kubernetes.io/projected/7f992cba-9c34-45a5-abc1-bfa307a8badb-kube-api-access-r5n9t\") pod \"payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w\" (UID: \"7f992cba-9c34-45a5-abc1-bfa307a8badb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:11.645494 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.645341    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f992cba-9c34-45a5-abc1-bfa307a8badb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w\" (UID: \"7f992cba-9c34-45a5-abc1-bfa307a8badb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:11.646460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.646434    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f992cba-9c34-45a5-abc1-bfa307a8badb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w\" (UID: \"7f992cba-9c34-45a5-abc1-bfa307a8badb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:11.672173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.672091    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5n9t\" (UniqueName: \"kubernetes.io/projected/7f992cba-9c34-45a5-abc1-bfa307a8badb-kube-api-access-r5n9t\") pod \"payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w\" (UID: \"7f992cba-9c34-45a5-abc1-bfa307a8badb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:11.751993 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.751918    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:23:11.752847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.752829    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:23:11.765039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:11.765017    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:12.066859 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:23:12.066835    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f992cba_9c34_45a5_abc1_bfa307a8badb.slice/crio-e70b9f218f24f32c5ed13d25bb8bf0cd6ffd44f826d69388b7ae216ac79a1e58 WatchSource:0}: Error finding container e70b9f218f24f32c5ed13d25bb8bf0cd6ffd44f826d69388b7ae216ac79a1e58: Status 404 returned error can't find the container with id e70b9f218f24f32c5ed13d25bb8bf0cd6ffd44f826d69388b7ae216ac79a1e58
Oct 30 13:23:12.067400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:12.067379    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"]
Oct 30 13:23:12.225670 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:12.225638    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w" event={"ID":"7f992cba-9c34-45a5-abc1-bfa307a8badb","Type":"ContainerStarted","Data":"3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058"}
Oct 30 13:23:12.225842 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:12.225695    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w" event={"ID":"7f992cba-9c34-45a5-abc1-bfa307a8badb","Type":"ContainerStarted","Data":"e70b9f218f24f32c5ed13d25bb8bf0cd6ffd44f826d69388b7ae216ac79a1e58"}
Oct 30 13:23:12.225931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:12.225913    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:12.270036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:12.269985    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w" podStartSLOduration=1.269973192 podStartE2EDuration="1.269973192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:12.269604821 +0000 UTC m=+3050.325098644" watchObservedRunningTime="2026-10-30 13:23:12.269973192 +0000 UTC m=+3050.325467016"
Oct 30 13:23:12.372758 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:23:12.372676    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:23:23.235779 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:23.235754    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:23:30.089130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.089101    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"]
Oct 30 13:23:30.094784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.094765    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.115160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.115134    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"]
Oct 30 13:23:30.187763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.187726    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\""
Oct 30 13:23:30.187946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.187825    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\""
Oct 30 13:23:30.187946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.187828    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\""
Oct 30 13:23:30.187946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.187842    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\""
Oct 30 13:23:30.198706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.198685    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzjq5\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-kube-api-access-zzjq5\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.198845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.198826    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.198890 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.198866    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.199164 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.199147    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-f0f7bfc91557\""
Oct 30 13:23:30.199274 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.199209    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.199399 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.199308    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-11b2d8954595\""
Oct 30 13:23:30.199483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.199468    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-2m475\""
Oct 30 13:23:30.300172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.300134    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.300399 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.300181    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.300399 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.300383    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.300540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.300477    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjq5\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-kube-api-access-zzjq5\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.301017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.300983    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.301143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.301085    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.303098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.303071    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.330343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.330317    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjq5\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-kube-api-access-zzjq5\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.409037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.408970    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:30.711439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:30.711414    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"]
Oct 30 13:23:30.711729 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:23:30.711704    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e34c95_fa74_4b5e_8ad0_1dd13466be57.slice/crio-e44b9243966505e0a251e8b9cd24b0daf83699c4c59bd19fb03639b2572b6046 WatchSource:0}: Error finding container e44b9243966505e0a251e8b9cd24b0daf83699c4c59bd19fb03639b2572b6046: Status 404 returned error can't find the container with id e44b9243966505e0a251e8b9cd24b0daf83699c4c59bd19fb03639b2572b6046
Oct 30 13:23:31.357454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:31.357412    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb" event={"ID":"a6e34c95-fa74-4b5e-8ad0-1dd13466be57","Type":"ContainerStarted","Data":"6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3"}
Oct 30 13:23:31.357940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:31.357502    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb" event={"ID":"a6e34c95-fa74-4b5e-8ad0-1dd13466be57","Type":"ContainerStarted","Data":"e44b9243966505e0a251e8b9cd24b0daf83699c4c59bd19fb03639b2572b6046"}
Oct 30 13:23:31.357940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:31.357657    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:31.404980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:31.404930    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb" podStartSLOduration=1.404917038 podStartE2EDuration="1.404917038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:31.403840173 +0000 UTC m=+3069.459333997" watchObservedRunningTime="2026-10-30 13:23:31.404917038 +0000 UTC m=+3069.460410861"
Oct 30 13:23:42.367850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:23:42.367813    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:23:42.372646 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:23:42.372613    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:24:12.373359 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:12.373266    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:24:30.061464 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.061430    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"]
Oct 30 13:24:30.066185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.066161    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.093274 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.093241    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"]
Oct 30 13:24:30.101303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.101256    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4db150\""
Oct 30 13:24:30.101470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.101313    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-e2e--1b2e61a1-dockercfg-cmc52\""
Oct 30 13:24:30.101470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.101392    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"agc-cred-911d586b1fd4\""
Oct 30 13:24:30.101592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.101472    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"routing-overlay\""
Oct 30 13:24:30.101592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.101502    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"kube-root-ca.crt\""
Oct 30 13:24:30.101723 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.101703    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"openshift-service-ca.crt\""
Oct 30 13:24:30.128330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.128275    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmnwb\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-kube-api-access-wmnwb\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.128561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.128396    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.128561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.128505    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.128697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.128575    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.230065 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.230028    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmnwb\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-kube-api-access-wmnwb\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.230265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.230076    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.230265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.230129    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.230265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.230188    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.231070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.230992    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.231070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.231067    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.232880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.232851    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.255327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.255267    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmnwb\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-kube-api-access-wmnwb\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.381894 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.381814    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:30.750477 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:24:30.750439    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52eb17fc_3ab4_4c28_a3bc_d0446bdb8cbf.slice/crio-b93a16fd214335a5e133769d0e692748f9de78cde6c08960919084e37e662ad9 WatchSource:0}: Error finding container b93a16fd214335a5e133769d0e692748f9de78cde6c08960919084e37e662ad9: Status 404 returned error can't find the container with id b93a16fd214335a5e133769d0e692748f9de78cde6c08960919084e37e662ad9
Oct 30 13:24:30.752004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.751954    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"]
Oct 30 13:24:30.803498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:30.803464    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h" event={"ID":"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf","Type":"ContainerStarted","Data":"b93a16fd214335a5e133769d0e692748f9de78cde6c08960919084e37e662ad9"}
Oct 30 13:24:31.813277 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:31.813244    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h" event={"ID":"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf","Type":"ContainerStarted","Data":"6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9"}
Oct 30 13:24:31.813739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:31.813359    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:31.865795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:31.865737    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h" podStartSLOduration=1.865723118 podStartE2EDuration="1.865723118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:31.859165473 +0000 UTC m=+3129.914659298" watchObservedRunningTime="2026-10-30 13:24:31.865723118 +0000 UTC m=+3129.921216942"
Oct 30 13:24:34.150499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.150462    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"]
Oct 30 13:24:34.155139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.155122    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.185054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.185019    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"agc-cred-55d30e1e72c7\""
Oct 30 13:24:34.185744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.185710    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"]
Oct 30 13:24:34.195841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.195812    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.195997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.195964    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.196164 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.196146    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzjz\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-kube-api-access-8bzjz\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.196242 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.196217    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.297348 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.297275    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.297548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.297420    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.297548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.297508    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.297702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.297686    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzjz\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-kube-api-access-8bzjz\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.297972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.297947    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.298399 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.298375    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.300136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.300118    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.323505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.323478    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzjz\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-kube-api-access-8bzjz\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.479132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.479098    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:34.787086 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:24:34.787054    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99ce267_85e6_469d_b680_a45296d3dd08.slice/crio-2804704e03694d04b4c4ffb3af3712de2ef20a61469be35aaa9f75cca8731598 WatchSource:0}: Error finding container 2804704e03694d04b4c4ffb3af3712de2ef20a61469be35aaa9f75cca8731598: Status 404 returned error can't find the container with id 2804704e03694d04b4c4ffb3af3712de2ef20a61469be35aaa9f75cca8731598
Oct 30 13:24:34.788073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.788046    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"]
Oct 30 13:24:34.838117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:34.838085    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" event={"ID":"d99ce267-85e6-469d-b680-a45296d3dd08","Type":"ContainerStarted","Data":"2804704e03694d04b4c4ffb3af3712de2ef20a61469be35aaa9f75cca8731598"}
Oct 30 13:24:35.847590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:35.847558    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" event={"ID":"d99ce267-85e6-469d-b680-a45296d3dd08","Type":"ContainerStarted","Data":"939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373"}
Oct 30 13:24:35.847983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:35.847769    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:35.896109 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:35.896056    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" podStartSLOduration=1.896041667 podStartE2EDuration="1.896041667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:35.895008582 +0000 UTC m=+3133.950502407" watchObservedRunningTime="2026-10-30 13:24:35.896041667 +0000 UTC m=+3133.951535496"
Oct 30 13:24:36.423796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:36.423764    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"]
Oct 30 13:24:36.426942 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:36.426918    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:36.946533 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:36.946505    4429 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-75d385/agc-cred-55d30e1e72c7: secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:36.946533 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:36.946533    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4: secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:36.947029 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:36.946592    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials podName:d99ce267-85e6-469d-b680-a45296d3dd08 nodeName:}" failed. No retries permitted until 2026-10-30 13:24:37.446572056 +0000 UTC m=+3135.502065863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" (UID: "d99ce267-85e6-469d-b680-a45296d3dd08") : secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:37.454673 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:37.454642    4429 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-75d385/agc-cred-55d30e1e72c7: secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:37.454673 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:37.454676    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4: secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:37.454890 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:37.454731    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials podName:d99ce267-85e6-469d-b680-a45296d3dd08 nodeName:}" failed. No retries permitted until 2026-10-30 13:24:38.454716719 +0000 UTC m=+3136.510210521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" (UID: "d99ce267-85e6-469d-b680-a45296d3dd08") : secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:37.728310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:37.728207    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"]
Oct 30 13:24:38.467738 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:38.467703    4429 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-75d385/agc-cred-55d30e1e72c7: secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:38.467738 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:38.467735    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4: secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:38.468176 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:38.467796    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials podName:d99ce267-85e6-469d-b680-a45296d3dd08 nodeName:}" failed. No retries permitted until 2026-10-30 13:24:40.46778209 +0000 UTC m=+3138.523275904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" (UID: "d99ce267-85e6-469d-b680-a45296d3dd08") : secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:40.496662 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:40.496626    4429 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-75d385/agc-cred-55d30e1e72c7: secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:40.496662 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:40.496662    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4: secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:40.497098 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:40.496737    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials podName:d99ce267-85e6-469d-b680-a45296d3dd08 nodeName:}" failed. No retries permitted until 2026-10-30 13:24:44.496716138 +0000 UTC m=+3142.552209942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" (UID: "d99ce267-85e6-469d-b680-a45296d3dd08") : secret "agc-cred-55d30e1e72c7" not found
Oct 30 13:24:41.424794 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.424735    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h" podUID="52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" containerName="payload-processing" containerID="cri-o://6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9" gracePeriod=25
Oct 30 13:24:41.703539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.703511    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:41.817439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.817407    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-plugins-config-volume\") pod \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") "
Oct 30 13:24:41.817610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.817462    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-routing-overlay-volume\") pod \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") "
Oct 30 13:24:41.817610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.817492    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-provider-credentials\") pod \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") "
Oct 30 13:24:41.817864 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.817844    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-kube-api-access-wmnwb\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-kube-api-access-wmnwb\") pod \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\" (UID: \"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf\") "
Oct 30 13:24:41.817922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.817870    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-plugins-config-volume" pod "52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" (UID: "52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:41.817922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.817877    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-routing-overlay-volume" pod "52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" (UID: "52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:41.818716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.818689    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:41.818858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.818720    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:41.820066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.820038    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-provider-credentials" pod "52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" (UID: "52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:41.820205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.820164    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-kube-api-access-wmnwb" pod "52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" (UID: "52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf"). InnerVolumeSpecName "kube-api-access-wmnwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:41.911102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.911064    4429 generic.go:356] "Generic (PLEG): container finished" podID="52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" containerID="6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9" exitCode=0
Oct 30 13:24:41.911273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.911115    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"
Oct 30 13:24:41.911273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.911153    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h" event={"ID":"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf","Type":"ContainerDied","Data":"6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9"}
Oct 30 13:24:41.911273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.911211    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h" event={"ID":"52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf","Type":"ContainerDied","Data":"b93a16fd214335a5e133769d0e692748f9de78cde6c08960919084e37e662ad9"}
Oct 30 13:24:41.911273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.911235    4429 scope.go:122] "RemoveContainer" containerID="6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9"
Oct 30 13:24:41.919859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.919836    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmnwb\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-kube-api-access-wmnwb\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:41.919982 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.919860    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:41.922639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.922626    4429 scope.go:122] "RemoveContainer" containerID="6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9"
Oct 30 13:24:41.923740 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:41.923715    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9\": container with ID starting with 6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9 not found: ID does not exist" containerID="6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9"
Oct 30 13:24:41.923818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.923751    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9"} err="failed to get container status \"6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9\": rpc error: code = NotFound desc = could not find container \"6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9\": container with ID starting with 6a5dd3ca59ae216e8db5c46ce07d46ed60f6d14776e2986229e2598a971b46f9 not found: ID does not exist"
Oct 30 13:24:41.970945 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.970920    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"]
Oct 30 13:24:41.974351 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:41.974328    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-f8c8kkz8h"]
Oct 30 13:24:42.077468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:42.077437    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" path="/var/lib/kubelet/pods/52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf/volumes"
Oct 30 13:24:42.373912 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:42.373838    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:24:42.420337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:42.420300    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"]
Oct 30 13:24:42.862704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:42.862654    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" podUID="d99ce267-85e6-469d-b680-a45296d3dd08" containerName="payload-processing" containerID="cri-o://939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373" gracePeriod=25
Oct 30 13:24:43.185921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.185889    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:43.238636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.238601    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-routing-overlay-volume\") pod \"d99ce267-85e6-469d-b680-a45296d3dd08\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") "
Oct 30 13:24:43.238768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.238665    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-plugins-config-volume\") pod \"d99ce267-85e6-469d-b680-a45296d3dd08\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") "
Oct 30 13:24:43.238768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.238696    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-kube-api-access-8bzjz\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-kube-api-access-8bzjz\") pod \"d99ce267-85e6-469d-b680-a45296d3dd08\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") "
Oct 30 13:24:43.238893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.238767    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials\") pod \"d99ce267-85e6-469d-b680-a45296d3dd08\" (UID: \"d99ce267-85e6-469d-b680-a45296d3dd08\") "
Oct 30 13:24:43.239664 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.239638    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-routing-overlay-volume" pod "d99ce267-85e6-469d-b680-a45296d3dd08" (UID: "d99ce267-85e6-469d-b680-a45296d3dd08"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:43.239802 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.239696    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-plugins-config-volume" pod "d99ce267-85e6-469d-b680-a45296d3dd08" (UID: "d99ce267-85e6-469d-b680-a45296d3dd08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:43.241866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.241844    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials" pod "d99ce267-85e6-469d-b680-a45296d3dd08" (UID: "d99ce267-85e6-469d-b680-a45296d3dd08"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:43.242295 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.242260    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-kube-api-access-8bzjz" pod "d99ce267-85e6-469d-b680-a45296d3dd08" (UID: "d99ce267-85e6-469d-b680-a45296d3dd08"). InnerVolumeSpecName "kube-api-access-8bzjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:43.340781 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.340747    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:43.340932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.340786    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:43.340932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.340802    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99ce267-85e6-469d-b680-a45296d3dd08-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:43.340932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.340830    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bzjz\" (UniqueName: \"kubernetes.io/projected/d99ce267-85e6-469d-b680-a45296d3dd08-kube-api-access-8bzjz\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:43.942515 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.942474    4429 generic.go:356] "Generic (PLEG): container finished" podID="d99ce267-85e6-469d-b680-a45296d3dd08" containerID="939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373" exitCode=0
Oct 30 13:24:43.942997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.942523    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"
Oct 30 13:24:43.942997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.942560    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" event={"ID":"d99ce267-85e6-469d-b680-a45296d3dd08","Type":"ContainerDied","Data":"939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373"}
Oct 30 13:24:43.942997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.942619    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4" event={"ID":"d99ce267-85e6-469d-b680-a45296d3dd08","Type":"ContainerDied","Data":"2804704e03694d04b4c4ffb3af3712de2ef20a61469be35aaa9f75cca8731598"}
Oct 30 13:24:43.942997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.942649    4429 scope.go:122] "RemoveContainer" containerID="939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373"
Oct 30 13:24:43.954767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.954747    4429 scope.go:122] "RemoveContainer" containerID="939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373"
Oct 30 13:24:43.955051 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:43.955036    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373\": container with ID starting with 939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373 not found: ID does not exist" containerID="939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373"
Oct 30 13:24:43.955111 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:43.955064    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373"} err="failed to get container status \"939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373\": rpc error: code = NotFound desc = could not find container \"939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373\": container with ID starting with 939753bcb60966dec3511866bec3c765f037f3b2437f196f0bc5be1b1f698373 not found: ID does not exist"
Oct 30 13:24:44.013632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:44.013606    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"]
Oct 30 13:24:44.026570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:44.026544    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-6645ljvd4"]
Oct 30 13:24:44.076400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:44.076377    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99ce267-85e6-469d-b680-a45296d3dd08" path="/var/lib/kubelet/pods/d99ce267-85e6-469d-b680-a45296d3dd08/volumes"
Oct 30 13:24:47.421322 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.421235    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb" podUID="a6e34c95-fa74-4b5e-8ad0-1dd13466be57" containerName="payload-processing" containerID="cri-o://6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3" gracePeriod=25
Oct 30 13:24:47.716601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.716576    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:24:47.800511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.800474    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-provider-credentials\") pod \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") "
Oct 30 13:24:47.800743 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.800720    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-routing-overlay-volume\") pod \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") "
Oct 30 13:24:47.800858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.800835    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-plugins-config-volume\") pod \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") "
Oct 30 13:24:47.801124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.801106    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-kube-api-access-zzjq5\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-kube-api-access-zzjq5\") pod \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\" (UID: \"a6e34c95-fa74-4b5e-8ad0-1dd13466be57\") "
Oct 30 13:24:47.801439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.801149    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-routing-overlay-volume" pod "a6e34c95-fa74-4b5e-8ad0-1dd13466be57" (UID: "a6e34c95-fa74-4b5e-8ad0-1dd13466be57"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:47.801538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.801194    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-plugins-config-volume" pod "a6e34c95-fa74-4b5e-8ad0-1dd13466be57" (UID: "a6e34c95-fa74-4b5e-8ad0-1dd13466be57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:47.801892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.801873    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:47.801972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.801897    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:47.803590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.803561    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-provider-credentials" pod "a6e34c95-fa74-4b5e-8ad0-1dd13466be57" (UID: "a6e34c95-fa74-4b5e-8ad0-1dd13466be57"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:47.808057 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.808029    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-kube-api-access-zzjq5" pod "a6e34c95-fa74-4b5e-8ad0-1dd13466be57" (UID: "a6e34c95-fa74-4b5e-8ad0-1dd13466be57"). InnerVolumeSpecName "kube-api-access-zzjq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:47.903117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.903078    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzjq5\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-kube-api-access-zzjq5\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:47.903117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.903118    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e34c95-fa74-4b5e-8ad0-1dd13466be57-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:47.987881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.987848    4429 generic.go:356] "Generic (PLEG): container finished" podID="a6e34c95-fa74-4b5e-8ad0-1dd13466be57" containerID="6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3" exitCode=0
Oct 30 13:24:47.988089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.987910    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"
Oct 30 13:24:47.988089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.987936    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb" event={"ID":"a6e34c95-fa74-4b5e-8ad0-1dd13466be57","Type":"ContainerDied","Data":"6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3"}
Oct 30 13:24:47.988089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.988028    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb" event={"ID":"a6e34c95-fa74-4b5e-8ad0-1dd13466be57","Type":"ContainerDied","Data":"e44b9243966505e0a251e8b9cd24b0daf83699c4c59bd19fb03639b2572b6046"}
Oct 30 13:24:47.988089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:47.988061    4429 scope.go:122] "RemoveContainer" containerID="6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3"
Oct 30 13:24:48.000946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:48.000924    4429 scope.go:122] "RemoveContainer" containerID="6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3"
Oct 30 13:24:48.001242 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:48.001223    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3\": container with ID starting with 6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3 not found: ID does not exist" containerID="6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3"
Oct 30 13:24:48.001313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:48.001250    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3"} err="failed to get container status \"6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3\": rpc error: code = NotFound desc = could not find container \"6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3\": container with ID starting with 6426f300e287913d4705a2b9815938cb3f92cbdec69556013e99a7ebcf1ac1b3 not found: ID does not exist"
Oct 30 13:24:48.049795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:48.049761    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"]
Oct 30 13:24:48.062722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:48.062698    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfvkfkb"]
Oct 30 13:24:48.076107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:48.076083    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e34c95-fa74-4b5e-8ad0-1dd13466be57" path="/var/lib/kubelet/pods/a6e34c95-fa74-4b5e-8ad0-1dd13466be57/volumes"
Oct 30 13:24:52.557548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:52.557515    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"]
Oct 30 13:24:57.558374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.558333    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg" podUID="17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" containerName="payload-processing" containerID="cri-o://7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7" gracePeriod=25
Oct 30 13:24:57.831460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.831436    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:24:57.931708 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.931680    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-kube-api-access-56qlz\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-kube-api-access-56qlz\") pod \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") "
Oct 30 13:24:57.931926 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.931908    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-provider-credentials\") pod \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") "
Oct 30 13:24:57.931965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.931946    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-routing-overlay-volume\") pod \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") "
Oct 30 13:24:57.932054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.932035    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-plugins-config-volume\") pod \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\" (UID: \"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6\") "
Oct 30 13:24:57.932376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.932348    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-routing-overlay-volume" pod "17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" (UID: "17a1dc4e-1f0e-4295-b2b3-f0193f7311f6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:57.932681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.932449    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-plugins-config-volume" pod "17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" (UID: "17a1dc4e-1f0e-4295-b2b3-f0193f7311f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:57.932886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.932860    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:57.932962 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.932892    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:57.934219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.934198    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-kube-api-access-56qlz" pod "17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" (UID: "17a1dc4e-1f0e-4295-b2b3-f0193f7311f6"). InnerVolumeSpecName "kube-api-access-56qlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:57.934332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:57.934264    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-provider-credentials" pod "17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" (UID: "17a1dc4e-1f0e-4295-b2b3-f0193f7311f6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:58.033980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.033946    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:58.033980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.033973    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56qlz\" (UniqueName: \"kubernetes.io/projected/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6-kube-api-access-56qlz\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:24:58.059512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.059413    4429 generic.go:356] "Generic (PLEG): container finished" podID="17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" containerID="7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7" exitCode=0
Oct 30 13:24:58.059512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.059469    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"
Oct 30 13:24:58.059725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.059462    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg" event={"ID":"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6","Type":"ContainerDied","Data":"7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7"}
Oct 30 13:24:58.059725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.059591    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg" event={"ID":"17a1dc4e-1f0e-4295-b2b3-f0193f7311f6","Type":"ContainerDied","Data":"f1ddac16698a147ee789e5d6093633a0de8a20c9241e04933d0a051e56a60c02"}
Oct 30 13:24:58.059725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.059620    4429 scope.go:122] "RemoveContainer" containerID="7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7"
Oct 30 13:24:58.073810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.073771    4429 scope.go:122] "RemoveContainer" containerID="7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7"
Oct 30 13:24:58.085110 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:24:58.085076    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7\": container with ID starting with 7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7 not found: ID does not exist" containerID="7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7"
Oct 30 13:24:58.085238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.085123    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7"} err="failed to get container status \"7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7\": rpc error: code = NotFound desc = could not find container \"7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7\": container with ID starting with 7af6374b60f78dc8d9ba24b1fda98017e63e5339fa6d265f199ae013274cb2f7 not found: ID does not exist"
Oct 30 13:24:58.126706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.126676    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"]
Oct 30 13:24:58.130819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:24:58.130794    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-8969cn9rg"]
Oct 30 13:25:00.076694 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:00.076653    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" path="/var/lib/kubelet/pods/17a1dc4e-1f0e-4295-b2b3-f0193f7311f6/volumes"
Oct 30 13:25:01.300756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.300725    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"]
Oct 30 13:25:01.301592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301575    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d99ce267-85e6-469d-b680-a45296d3dd08" containerName="payload-processing"
Oct 30 13:25:01.301592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301592    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99ce267-85e6-469d-b680-a45296d3dd08" containerName="payload-processing"
Oct 30 13:25:01.301810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301612    4429 cpu_manager.go:414] "containerMap: removing container" podUID="52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" containerName="payload-processing"
Oct 30 13:25:01.301810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301618    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" containerName="payload-processing"
Oct 30 13:25:01.301810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301636    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a6e34c95-fa74-4b5e-8ad0-1dd13466be57" containerName="payload-processing"
Oct 30 13:25:01.301810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301641    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e34c95-fa74-4b5e-8ad0-1dd13466be57" containerName="payload-processing"
Oct 30 13:25:01.301810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301691    4429 cpu_manager.go:414] "containerMap: removing container" podUID="17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" containerName="payload-processing"
Oct 30 13:25:01.301810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301697    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" containerName="payload-processing"
Oct 30 13:25:01.301810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301810    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a1dc4e-1f0e-4295-b2b3-f0193f7311f6" containerName="payload-processing"
Oct 30 13:25:01.302080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301819    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99ce267-85e6-469d-b680-a45296d3dd08" containerName="payload-processing"
Oct 30 13:25:01.302080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301826    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="52eb17fc-3ab4-4c28-a3bc-d0446bdb8cbf" containerName="payload-processing"
Oct 30 13:25:01.302080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.301834    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e34c95-fa74-4b5e-8ad0-1dd13466be57" containerName="payload-processing"
Oct 30 13:25:01.306428 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.306410    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.328259 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.328236    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"]
Oct 30 13:25:01.337925 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.337900    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-75d385-openshift-default-dockercfg-fhdv5\""
Oct 30 13:25:01.479273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479233    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdsg\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-kube-api-access-jkdsg\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.479462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479313    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-credential-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.479462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479385    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-token\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.479462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479427    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-certs\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.479462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479448    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-data\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.479605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479582    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-envoy\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.479644 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479626    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.479678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479654    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istiod-ca-cert\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.479725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.479707    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-podinfo\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.580609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580537    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.580609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580572    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istiod-ca-cert\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.580609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580598    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-podinfo\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.580825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580727    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdsg\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-kube-api-access-jkdsg\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.580825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580753    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-credential-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.580825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580773    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-token\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.580825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580802    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-certs\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.580825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580820    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-data\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.581093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.580875    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-envoy\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.581093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.581001    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.581209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.581136    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-credential-socket\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.581209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.581197    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-certs\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.581356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.581315    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-data\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.581833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.581806    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istiod-ca-cert\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.583469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.583443    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-envoy\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.583583 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.583561    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-podinfo\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.606395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.606373    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-token\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.606496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.606447    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdsg\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-kube-api-access-jkdsg\") pod \"e2e-extmod-b-75d385-openshift-default-8785c777c-zt757\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") " pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.620400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.620376    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:01.945374 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:25:01.945337    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc56bee0_eb57_46f5_b67a_2fa923d073c2.slice/crio-66a050042bedc6fe2d1ebfa9a0dfb23abaa1fd20b7278f68099dfe0eb0f8e124 WatchSource:0}: Error finding container 66a050042bedc6fe2d1ebfa9a0dfb23abaa1fd20b7278f68099dfe0eb0f8e124: Status 404 returned error can't find the container with id 66a050042bedc6fe2d1ebfa9a0dfb23abaa1fd20b7278f68099dfe0eb0f8e124
Oct 30 13:25:01.945816 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.945796    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"]
Oct 30 13:25:01.948601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.948572    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:01.948692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.948633    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:01.948692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:01.948662    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:02.090648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:02.090612    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" event={"ID":"bc56bee0-eb57-46f5-b67a-2fa923d073c2","Type":"ContainerStarted","Data":"532d887086c7700d6ec37fe02f59caa329da2e67d9c208bdf2aa78de324178b4"}
Oct 30 13:25:02.090869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:02.090679    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" event={"ID":"bc56bee0-eb57-46f5-b67a-2fa923d073c2","Type":"ContainerStarted","Data":"66a050042bedc6fe2d1ebfa9a0dfb23abaa1fd20b7278f68099dfe0eb0f8e124"}
Oct 30 13:25:02.140226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:02.140165    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" podStartSLOduration=1.140147351 podStartE2EDuration="1.140147351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:02.139194709 +0000 UTC m=+3160.194688548" watchObservedRunningTime="2026-10-30 13:25:02.140147351 +0000 UTC m=+3160.195641178"
Oct 30 13:25:02.621193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:02.621164    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:03.622378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:03.622343    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:25:03.622864 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:03.622409    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:25:04.621117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:04.621086    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:25:04.621307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:04.621152    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:25:05.621531 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:05.621488    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:25:05.622001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:05.621572    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:25:06.620983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:06.620945    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:25:06.621174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:06.621007    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:25:07.626208 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:07.626171    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:25:07.627502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:07.627471    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:25:08.627187 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:08.627159    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:08.627652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:08.627632    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:08.628449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:08.628427    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:25:12.373978 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:25:12.373943    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:25:15.574527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.571822    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"]
Oct 30 13:25:15.584143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.584117    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.597473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.597435    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"]
Oct 30 13:25:15.607039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.607009    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-63499e-openshift-default-dockercfg-77r7m\""
Oct 30 13:25:15.702475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702441    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.702663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702584    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-envoy\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.702748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702692    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a128d702-ca8a-455f-8ec0-42244bba082b-istio-podinfo\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.702808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702766    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcgl\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-kube-api-access-kgcgl\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.702808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702795    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-data\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.702915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702898    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a128d702-ca8a-455f-8ec0-42244bba082b-istiod-ca-cert\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.702973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702945    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-credential-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.703031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702981    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-certs\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.703031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.702997    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-istio-token\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.804873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.804836    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-data\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.804899    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a128d702-ca8a-455f-8ec0-42244bba082b-istiod-ca-cert\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.804926    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-credential-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.804965    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-certs\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.804989    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-istio-token\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805104    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805183    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-envoy\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805257    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a128d702-ca8a-455f-8ec0-42244bba082b-istio-podinfo\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805320    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-data\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805358    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-certs\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805387    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcgl\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-kube-api-access-kgcgl\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805477    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-credential-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805582    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-socket\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.805801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.805773    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a128d702-ca8a-455f-8ec0-42244bba082b-istiod-ca-cert\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.807867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.807850    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-envoy\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.808005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.807988    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a128d702-ca8a-455f-8ec0-42244bba082b-istio-podinfo\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.831345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.831258    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-istio-token\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.831673 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.831656    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcgl\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-kube-api-access-kgcgl\") pod \"e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") " pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:15.906342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:15.906312    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:16.257961 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:25:16.257927    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda128d702_ca8a_455f_8ec0_42244bba082b.slice/crio-65789f3e847d254a89ccb6839f74c5f632ba466b4e59060d13b160917c2a476a WatchSource:0}: Error finding container 65789f3e847d254a89ccb6839f74c5f632ba466b4e59060d13b160917c2a476a: Status 404 returned error can't find the container with id 65789f3e847d254a89ccb6839f74c5f632ba466b4e59060d13b160917c2a476a
Oct 30 13:25:16.259099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:16.259071    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"]
Oct 30 13:25:16.261055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:16.261020    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:16.261141 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:16.261093    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:16.261141 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:16.261122    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:17.196828 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:17.196790    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" event={"ID":"a128d702-ca8a-455f-8ec0-42244bba082b","Type":"ContainerStarted","Data":"e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498"}
Oct 30 13:25:17.197269 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:17.196853    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" event={"ID":"a128d702-ca8a-455f-8ec0-42244bba082b","Type":"ContainerStarted","Data":"65789f3e847d254a89ccb6839f74c5f632ba466b4e59060d13b160917c2a476a"}
Oct 30 13:25:17.285326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:17.284534    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" podStartSLOduration=2.284505442 podStartE2EDuration="2.284505442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:17.263246219 +0000 UTC m=+3175.318740070" watchObservedRunningTime="2026-10-30 13:25:17.284505442 +0000 UTC m=+3175.339999305"
Oct 30 13:25:17.907140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:17.907103    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:17.908961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:17.908939    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:25:17.909042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:17.908987    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:25:18.906984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:18.906947    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:25:18.907434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:18.907009    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:25:19.907210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:19.907165    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:25:19.907754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:19.907231    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:25:20.906907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:20.906868    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:25:20.907115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:20.906925    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:25:21.907080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:21.907037    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:25:21.907561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:21.907092    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:25:22.912910 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:22.912881    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:22.913384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:22.913330    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:22.914252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:22.914233    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:25:34.078624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.078556    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"]
Oct 30 13:25:34.084585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.084566    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.098834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.098803    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"]
Oct 30 13:25:34.107061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.107015    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\""
Oct 30 13:25:34.107628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.107606    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-f0f7bfc91557\""
Oct 30 13:25:34.107834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.107670    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\""
Oct 30 13:25:34.107834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.107747    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-2m475\""
Oct 30 13:25:34.115047 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.115024    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-11b2d8954595\""
Oct 30 13:25:34.115183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.115030    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\""
Oct 30 13:25:34.115183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.115049    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\""
Oct 30 13:25:34.203405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.203366    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.203591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.203505    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4rlb\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-kube-api-access-s4rlb\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.203591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.203588    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.203693 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.203671    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.305312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.305235    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4rlb\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-kube-api-access-s4rlb\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.305506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.305346    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.305506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.305399    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.305587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.305548    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.306345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.306230    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.306345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.306272    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.308228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.308206    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.330564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.330490    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4rlb\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-kube-api-access-s4rlb\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.401541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.401497    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:34.710517 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:25:34.710475    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd046a271_d20b_4a6e_9eba_f6518d634e00.slice/crio-9b5c055188f290def6f0ce55980b26fee45037ea7627151356336ef12fcb669e WatchSource:0}: Error finding container 9b5c055188f290def6f0ce55980b26fee45037ea7627151356336ef12fcb669e: Status 404 returned error can't find the container with id 9b5c055188f290def6f0ce55980b26fee45037ea7627151356336ef12fcb669e
Oct 30 13:25:34.712542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:34.712518    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"]
Oct 30 13:25:35.338383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:35.338342    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62" event={"ID":"d046a271-d20b-4a6e-9eba-f6518d634e00","Type":"ContainerStarted","Data":"42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34"}
Oct 30 13:25:35.338765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:35.338397    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62" event={"ID":"d046a271-d20b-4a6e-9eba-f6518d634e00","Type":"ContainerStarted","Data":"9b5c055188f290def6f0ce55980b26fee45037ea7627151356336ef12fcb669e"}
Oct 30 13:25:35.338765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:35.338436    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:35.388061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:35.388012    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62" podStartSLOduration=1.387998455 podStartE2EDuration="1.387998455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:35.385600078 +0000 UTC m=+3193.441093902" watchObservedRunningTime="2026-10-30 13:25:35.387998455 +0000 UTC m=+3193.443492278"
Oct 30 13:25:42.389900 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:25:42.389853    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:25:46.349064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:46.349036    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:25:47.990485 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:47.990448    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"]
Oct 30 13:25:47.997529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:47.997504    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.013091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.013056    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"]
Oct 30 13:25:48.106758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.106716    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mktc6\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-kube-api-access-mktc6\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.106962 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.106942    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.107103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.107087    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.107145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.107116    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.208395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.208353    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.208611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.208511    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.208611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.208549    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.208743 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.208622    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mktc6\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-kube-api-access-mktc6\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.209375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.209341    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.209474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.209341    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.211146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.211123    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.233657 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.233631    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mktc6\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-kube-api-access-mktc6\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.317159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.317058    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:48.626246 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:25:48.626204    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ac1d64_2618_4822_8fc5_e8a7bca63d02.slice/crio-e4dbdd2bfbf32649e6717e640733c15ebef534585f978c5fd04de80d7527e996 WatchSource:0}: Error finding container e4dbdd2bfbf32649e6717e640733c15ebef534585f978c5fd04de80d7527e996: Status 404 returned error can't find the container with id e4dbdd2bfbf32649e6717e640733c15ebef534585f978c5fd04de80d7527e996
Oct 30 13:25:48.626416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:48.626385    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"]
Oct 30 13:25:49.464147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:49.464111    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk" event={"ID":"f0ac1d64-2618-4822-8fc5-e8a7bca63d02","Type":"ContainerStarted","Data":"7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6"}
Oct 30 13:25:49.464577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:49.464170    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk" event={"ID":"f0ac1d64-2618-4822-8fc5-e8a7bca63d02","Type":"ContainerStarted","Data":"e4dbdd2bfbf32649e6717e640733c15ebef534585f978c5fd04de80d7527e996"}
Oct 30 13:25:49.464577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:49.464197    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:25:49.512988 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:49.512925    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk" podStartSLOduration=2.512907995 podStartE2EDuration="2.512907995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:49.509347186 +0000 UTC m=+3207.564841009" watchObservedRunningTime="2026-10-30 13:25:49.512907995 +0000 UTC m=+3207.568401820"
Oct 30 13:25:57.637390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:57.637351    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"]
Oct 30 13:25:57.639951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:25:57.639929    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:26:02.638246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:02.638202    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk" podUID="f0ac1d64-2618-4822-8fc5-e8a7bca63d02" containerName="payload-processing" containerID="cri-o://7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6" gracePeriod=25
Oct 30 13:26:02.945410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:02.945364    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:26:03.008539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.008499    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-plugins-config-volume\") pod \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") "
Oct 30 13:26:03.008539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.008546    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-provider-credentials\") pod \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") "
Oct 30 13:26:03.008815 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.008744    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-kube-api-access-mktc6\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-kube-api-access-mktc6\") pod \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") "
Oct 30 13:26:03.008888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.008846    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-routing-overlay-volume\") pod \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\" (UID: \"f0ac1d64-2618-4822-8fc5-e8a7bca63d02\") "
Oct 30 13:26:03.009052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.009024    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-plugins-config-volume" pod "f0ac1d64-2618-4822-8fc5-e8a7bca63d02" (UID: "f0ac1d64-2618-4822-8fc5-e8a7bca63d02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:03.009384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.009358    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-routing-overlay-volume" pod "f0ac1d64-2618-4822-8fc5-e8a7bca63d02" (UID: "f0ac1d64-2618-4822-8fc5-e8a7bca63d02"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:03.009847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.009825    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:26:03.009978 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.009851    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:26:03.011232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.011208    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-provider-credentials" pod "f0ac1d64-2618-4822-8fc5-e8a7bca63d02" (UID: "f0ac1d64-2618-4822-8fc5-e8a7bca63d02"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:03.011389 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.011254    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-kube-api-access-mktc6" pod "f0ac1d64-2618-4822-8fc5-e8a7bca63d02" (UID: "f0ac1d64-2618-4822-8fc5-e8a7bca63d02"). InnerVolumeSpecName "kube-api-access-mktc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:03.111368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.111337    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:26:03.111541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.111379    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mktc6\" (UniqueName: \"kubernetes.io/projected/f0ac1d64-2618-4822-8fc5-e8a7bca63d02-kube-api-access-mktc6\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:26:03.565897 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.565852    4429 generic.go:356] "Generic (PLEG): container finished" podID="f0ac1d64-2618-4822-8fc5-e8a7bca63d02" containerID="7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6" exitCode=0
Oct 30 13:26:03.566095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.565973    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk" event={"ID":"f0ac1d64-2618-4822-8fc5-e8a7bca63d02","Type":"ContainerDied","Data":"7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6"}
Oct 30 13:26:03.566095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.566090    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk" event={"ID":"f0ac1d64-2618-4822-8fc5-e8a7bca63d02","Type":"ContainerDied","Data":"e4dbdd2bfbf32649e6717e640733c15ebef534585f978c5fd04de80d7527e996"}
Oct 30 13:26:03.566195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.566115    4429 scope.go:122] "RemoveContainer" containerID="7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6"
Oct 30 13:26:03.566411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.566381    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"
Oct 30 13:26:03.577565 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.577549    4429 scope.go:122] "RemoveContainer" containerID="7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6"
Oct 30 13:26:03.577822 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:26:03.577806    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6\": container with ID starting with 7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6 not found: ID does not exist" containerID="7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6"
Oct 30 13:26:03.577861 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.577829    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6"} err="failed to get container status \"7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6\": rpc error: code = NotFound desc = could not find container \"7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6\": container with ID starting with 7102cb212b8b22051c40ae75a5a250b86831fe027c7e641bcac8ead0b1c0acc6 not found: ID does not exist"
Oct 30 13:26:03.625063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.625034    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"]
Oct 30 13:26:03.629206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:03.629177    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb42x8vk"]
Oct 30 13:26:04.076185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:04.076150    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ac1d64-2618-4822-8fc5-e8a7bca63d02" path="/var/lib/kubelet/pods/f0ac1d64-2618-4822-8fc5-e8a7bca63d02/volumes"
Oct 30 13:26:05.251923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.251889    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"]
Oct 30 13:26:05.252662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.252647    4429 cpu_manager.go:414] "containerMap: removing container" podUID="f0ac1d64-2618-4822-8fc5-e8a7bca63d02" containerName="payload-processing"
Oct 30 13:26:05.252711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.252663    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ac1d64-2618-4822-8fc5-e8a7bca63d02" containerName="payload-processing"
Oct 30 13:26:05.252796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.252787    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ac1d64-2618-4822-8fc5-e8a7bca63d02" containerName="payload-processing"
Oct 30 13:26:05.257485 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.257467    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.280984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.280955    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"]
Oct 30 13:26:05.288998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.288969    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"openshift-service-ca.crt\""
Oct 30 13:26:05.289185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.288969    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8740d4\""
Oct 30 13:26:05.289272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.289052    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"routing-overlay\""
Oct 30 13:26:05.290544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.290525    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"kube-root-ca.crt\""
Oct 30 13:26:05.290694 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.290526    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-d8628f05d549\""
Oct 30 13:26:05.290822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.290569    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-091b9769f7e9\""
Oct 30 13:26:05.293036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.292994    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-6e5458768ddf\""
Oct 30 13:26:05.293036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.293046    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-a4617c99fc7d\""
Oct 30 13:26:05.293224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.293085    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-e2e--77992003-dockercfg-zp5z8\""
Oct 30 13:26:05.345572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.345544    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.345757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.345617    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.345757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.345716    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.345884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.345765    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hlg\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-kube-api-access-55hlg\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.446958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.446923    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.447136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.447027    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.447136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.447066    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.447136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.447110    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hlg\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-kube-api-access-55hlg\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.447748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.447727    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.447866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.447721    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.449847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.449826    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.473019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.472994    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hlg\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-kube-api-access-55hlg\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.571797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.571722    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:05.879418 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:26:05.879383    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b48d7d_37e0_48db_a646_68e673f134f6.slice/crio-742618db85576350e04f23712e2075f2be30e6a0f4185d7bfabfa52b071e48cd WatchSource:0}: Error finding container 742618db85576350e04f23712e2075f2be30e6a0f4185d7bfabfa52b071e48cd: Status 404 returned error can't find the container with id 742618db85576350e04f23712e2075f2be30e6a0f4185d7bfabfa52b071e48cd
Oct 30 13:26:05.880865 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:05.880837    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"]
Oct 30 13:26:06.603562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:06.603529    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm" event={"ID":"54b48d7d-37e0-48db-a646-68e673f134f6","Type":"ContainerStarted","Data":"b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea"}
Oct 30 13:26:06.603952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:06.603592    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm" event={"ID":"54b48d7d-37e0-48db-a646-68e673f134f6","Type":"ContainerStarted","Data":"742618db85576350e04f23712e2075f2be30e6a0f4185d7bfabfa52b071e48cd"}
Oct 30 13:26:06.603952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:06.603622    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:06.651031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:06.650981    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm" podStartSLOduration=1.650967085 podStartE2EDuration="1.650967085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:06.649404786 +0000 UTC m=+3224.704898628" watchObservedRunningTime="2026-10-30 13:26:06.650967085 +0000 UTC m=+3224.706460916"
Oct 30 13:26:10.306449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:10.306412    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"]
Oct 30 13:26:12.375117 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:26:12.375068    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:26:15.306758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.306716    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62" podUID="d046a271-d20b-4a6e-9eba-f6518d634e00" containerName="payload-processing" containerID="cri-o://42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34" gracePeriod=25
Oct 30 13:26:15.584933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.584909    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:26:15.671198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.671156    4429 generic.go:356] "Generic (PLEG): container finished" podID="d046a271-d20b-4a6e-9eba-f6518d634e00" containerID="42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34" exitCode=0
Oct 30 13:26:15.671425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.671211    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"
Oct 30 13:26:15.671511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.671420    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62" event={"ID":"d046a271-d20b-4a6e-9eba-f6518d634e00","Type":"ContainerDied","Data":"42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34"}
Oct 30 13:26:15.671511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.671476    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62" event={"ID":"d046a271-d20b-4a6e-9eba-f6518d634e00","Type":"ContainerDied","Data":"9b5c055188f290def6f0ce55980b26fee45037ea7627151356336ef12fcb669e"}
Oct 30 13:26:15.671648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.671511    4429 scope.go:122] "RemoveContainer" containerID="42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34"
Oct 30 13:26:15.682757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.682706    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-routing-overlay-volume\") pod \"d046a271-d20b-4a6e-9eba-f6518d634e00\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") "
Oct 30 13:26:15.682995 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.682973    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-plugins-config-volume\") pod \"d046a271-d20b-4a6e-9eba-f6518d634e00\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") "
Oct 30 13:26:15.683172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.683074    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-provider-credentials\") pod \"d046a271-d20b-4a6e-9eba-f6518d634e00\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") "
Oct 30 13:26:15.683172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.683104    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-kube-api-access-s4rlb\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-kube-api-access-s4rlb\") pod \"d046a271-d20b-4a6e-9eba-f6518d634e00\" (UID: \"d046a271-d20b-4a6e-9eba-f6518d634e00\") "
Oct 30 13:26:15.683306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.683192    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-routing-overlay-volume" pod "d046a271-d20b-4a6e-9eba-f6518d634e00" (UID: "d046a271-d20b-4a6e-9eba-f6518d634e00"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:15.683646 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.683622    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-plugins-config-volume" pod "d046a271-d20b-4a6e-9eba-f6518d634e00" (UID: "d046a271-d20b-4a6e-9eba-f6518d634e00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:15.683755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.683719    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:26:15.683755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.683735    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d046a271-d20b-4a6e-9eba-f6518d634e00-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:26:15.685355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.685337    4429 scope.go:122] "RemoveContainer" containerID="42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34"
Oct 30 13:26:15.685752 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:26:15.685721    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34\": container with ID starting with 42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34 not found: ID does not exist" containerID="42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34"
Oct 30 13:26:15.685853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.685764    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34"} err="failed to get container status \"42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34\": rpc error: code = NotFound desc = could not find container \"42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34\": container with ID starting with 42a73a182d1a05d94d56b1c67336a2819040907fbbdbaa13953fc14117bb5b34 not found: ID does not exist"
Oct 30 13:26:15.685919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.685849    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-provider-credentials" pod "d046a271-d20b-4a6e-9eba-f6518d634e00" (UID: "d046a271-d20b-4a6e-9eba-f6518d634e00"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:15.686334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.686310    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-kube-api-access-s4rlb" pod "d046a271-d20b-4a6e-9eba-f6518d634e00" (UID: "d046a271-d20b-4a6e-9eba-f6518d634e00"). InnerVolumeSpecName "kube-api-access-s4rlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:15.784725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.784686    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:26:15.784725 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:15.784715    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4rlb\" (UniqueName: \"kubernetes.io/projected/d046a271-d20b-4a6e-9eba-f6518d634e00-kube-api-access-s4rlb\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:26:16.027857 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:16.027823    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"]
Oct 30 13:26:16.040932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:16.040906    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-67dfbzc62"]
Oct 30 13:26:16.076245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:16.076220    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d046a271-d20b-4a6e-9eba-f6518d634e00" path="/var/lib/kubelet/pods/d046a271-d20b-4a6e-9eba-f6518d634e00/volumes"
Oct 30 13:26:17.615481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:17.615452    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:26:42.373250 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:26:42.373212    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:26:56.471794 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.471760    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"]
Oct 30 13:26:56.472600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.472584    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d046a271-d20b-4a6e-9eba-f6518d634e00" containerName="payload-processing"
Oct 30 13:26:56.472663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.472601    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d046a271-d20b-4a6e-9eba-f6518d634e00" containerName="payload-processing"
Oct 30 13:26:56.472733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.472723    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d046a271-d20b-4a6e-9eba-f6518d634e00" containerName="payload-processing"
Oct 30 13:26:56.476659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.476636    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.499360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.499332    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"]
Oct 30 13:26:56.507897 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.507865    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"agc-cred-186229086c7f\""
Oct 30 13:26:56.508049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.507921    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0505b0\""
Oct 30 13:26:56.508049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.507930    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"routing-overlay\""
Oct 30 13:26:56.508049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.508006    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"kube-root-ca.crt\""
Oct 30 13:26:56.510216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.510178    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"payload-processing-external-model-e2e--d83bf907-dockercfg-cxtkv\""
Oct 30 13:26:56.510374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.510258    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"openshift-service-ca.crt\""
Oct 30 13:26:56.510374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.510278    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0505b0\"/\"agc-cred-fafbf0960658\""
Oct 30 13:26:56.569972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.569944    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.570143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.570005    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.570222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.570203    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpx2\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-kube-api-access-qfpx2\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.570337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.570316    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.671727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.671681    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.671928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.671854    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.671928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.671903    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.672061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.672036    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpx2\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-kube-api-access-qfpx2\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.672677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.672653    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.672821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.672733    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.674665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.674639    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.697447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.697422    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpx2\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-kube-api-access-qfpx2\") pod \"payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") " pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:56.791741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:56.791662    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:57.120822 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:26:57.120790    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b02ed48_e42c_49d6_9383_f8f73690a947.slice/crio-1cf4fc2b438c73eb58f668cdd6cc75014c087feec94634d40a1472452811163f WatchSource:0}: Error finding container 1cf4fc2b438c73eb58f668cdd6cc75014c087feec94634d40a1472452811163f: Status 404 returned error can't find the container with id 1cf4fc2b438c73eb58f668cdd6cc75014c087feec94634d40a1472452811163f
Oct 30 13:26:57.122099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:57.122063    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"]
Oct 30 13:26:57.990300 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:57.990244    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s" event={"ID":"0b02ed48-e42c-49d6-9383-f8f73690a947","Type":"ContainerStarted","Data":"4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83"}
Oct 30 13:26:57.990766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:57.990371    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s" event={"ID":"0b02ed48-e42c-49d6-9383-f8f73690a947","Type":"ContainerStarted","Data":"1cf4fc2b438c73eb58f668cdd6cc75014c087feec94634d40a1472452811163f"}
Oct 30 13:26:57.990766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:57.990536    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:26:58.038474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:26:58.038434    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s" podStartSLOduration=2.038421629 podStartE2EDuration="2.038421629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:58.036580927 +0000 UTC m=+3276.092074749" watchObservedRunningTime="2026-10-30 13:26:58.038421629 +0000 UTC m=+3276.093915453"
Oct 30 13:27:09.001093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:09.001019    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:27:12.372569 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:27:12.372529    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:27:18.412389 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:18.412359    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"]
Oct 30 13:27:23.413670 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.413617    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm" podUID="54b48d7d-37e0-48db-a646-68e673f134f6" containerName="payload-processing" containerID="cri-o://b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea" gracePeriod=25
Oct 30 13:27:23.705865 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.705838    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:27:23.771419 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.771391    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-kube-api-access-55hlg\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-kube-api-access-55hlg\") pod \"54b48d7d-37e0-48db-a646-68e673f134f6\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") "
Oct 30 13:27:23.771628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.771440    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-provider-credentials\") pod \"54b48d7d-37e0-48db-a646-68e673f134f6\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") "
Oct 30 13:27:23.771628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.771544    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-plugins-config-volume\") pod \"54b48d7d-37e0-48db-a646-68e673f134f6\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") "
Oct 30 13:27:23.771628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.771560    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-routing-overlay-volume\") pod \"54b48d7d-37e0-48db-a646-68e673f134f6\" (UID: \"54b48d7d-37e0-48db-a646-68e673f134f6\") "
Oct 30 13:27:23.772188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.772032    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-plugins-config-volume" pod "54b48d7d-37e0-48db-a646-68e673f134f6" (UID: "54b48d7d-37e0-48db-a646-68e673f134f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:23.772338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.772227    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-routing-overlay-volume" pod "54b48d7d-37e0-48db-a646-68e673f134f6" (UID: "54b48d7d-37e0-48db-a646-68e673f134f6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:23.772711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.772689    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:27:23.772813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.772718    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b48d7d-37e0-48db-a646-68e673f134f6-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:27:23.774120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.774094    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-kube-api-access-55hlg" pod "54b48d7d-37e0-48db-a646-68e673f134f6" (UID: "54b48d7d-37e0-48db-a646-68e673f134f6"). InnerVolumeSpecName "kube-api-access-55hlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:23.774214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.774126    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-provider-credentials" pod "54b48d7d-37e0-48db-a646-68e673f134f6" (UID: "54b48d7d-37e0-48db-a646-68e673f134f6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:23.874700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.874671    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55hlg\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-kube-api-access-55hlg\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:27:23.874700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:23.874698    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b48d7d-37e0-48db-a646-68e673f134f6-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:27:24.171126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.171079    4429 generic.go:356] "Generic (PLEG): container finished" podID="54b48d7d-37e0-48db-a646-68e673f134f6" containerID="b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea" exitCode=0
Oct 30 13:27:24.171438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.171222    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"
Oct 30 13:27:24.171438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.171237    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm" event={"ID":"54b48d7d-37e0-48db-a646-68e673f134f6","Type":"ContainerDied","Data":"b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea"}
Oct 30 13:27:24.171438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.171402    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm" event={"ID":"54b48d7d-37e0-48db-a646-68e673f134f6","Type":"ContainerDied","Data":"742618db85576350e04f23712e2075f2be30e6a0f4185d7bfabfa52b071e48cd"}
Oct 30 13:27:24.171636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.171544    4429 scope.go:122] "RemoveContainer" containerID="b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea"
Oct 30 13:27:24.183815 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.183798    4429 scope.go:122] "RemoveContainer" containerID="b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea"
Oct 30 13:27:24.184109 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:27:24.184089    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea\": container with ID starting with b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea not found: ID does not exist" containerID="b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea"
Oct 30 13:27:24.184168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.184116    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea"} err="failed to get container status \"b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea\": rpc error: code = NotFound desc = could not find container \"b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea\": container with ID starting with b168fd0913ae9873ef01fe961f33c2d30e2ad236c898c5ac606234b4e13fe3ea not found: ID does not exist"
Oct 30 13:27:24.233883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.233849    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"]
Oct 30 13:27:24.237744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:24.237724    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-67f4rfdzm"]
Oct 30 13:27:26.077296 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:26.077261    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b48d7d-37e0-48db-a646-68e673f134f6" path="/var/lib/kubelet/pods/54b48d7d-37e0-48db-a646-68e673f134f6/volumes"
Oct 30 13:27:42.375941 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:27:42.375902    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:27:42.507305 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.507261    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"]
Oct 30 13:27:42.508039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.508023    4429 cpu_manager.go:414] "containerMap: removing container" podUID="54b48d7d-37e0-48db-a646-68e673f134f6" containerName="payload-processing"
Oct 30 13:27:42.508039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.508039    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b48d7d-37e0-48db-a646-68e673f134f6" containerName="payload-processing"
Oct 30 13:27:42.508166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.508156    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b48d7d-37e0-48db-a646-68e673f134f6" containerName="payload-processing"
Oct 30 13:27:42.512969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.512948    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.532151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.532117    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.532338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.532169    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.532338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.532327    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.532572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.532530    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ktx\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-kube-api-access-45ktx\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.533592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.533569    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"]
Oct 30 13:27:42.542553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.542474    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-f0f7bfc91557\""
Oct 30 13:27:42.542704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.542655    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\""
Oct 30 13:27:42.542704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.542683    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\""
Oct 30 13:27:42.542985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.542964    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\""
Oct 30 13:27:42.544458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.544438    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\""
Oct 30 13:27:42.544562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.544481    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-2m475\""
Oct 30 13:27:42.633686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.633595    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ktx\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-kube-api-access-45ktx\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.633842 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.633719    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.633842 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.633744    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.633941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.633890    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.634393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.634371    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.634546 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.634530    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.636540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.636516    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.662689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.662658    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ktx\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-kube-api-access-45ktx\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:42.828204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:42.828165    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:43.148510 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:27:43.148478    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07f9c3c_b503_499b_a1d6_7e8b99698b00.slice/crio-cfc788fda0b72f1b0b41154a62558fe3f566d1c5de9b62ccb24ff2f7f53793e4 WatchSource:0}: Error finding container cfc788fda0b72f1b0b41154a62558fe3f566d1c5de9b62ccb24ff2f7f53793e4: Status 404 returned error can't find the container with id cfc788fda0b72f1b0b41154a62558fe3f566d1c5de9b62ccb24ff2f7f53793e4
Oct 30 13:27:43.149695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:43.149658    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"]
Oct 30 13:27:43.320231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:43.320188    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4" event={"ID":"b07f9c3c-b503-499b-a1d6-7e8b99698b00","Type":"ContainerStarted","Data":"f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741"}
Oct 30 13:27:43.320456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:43.320267    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:27:43.320456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:43.320312    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4" event={"ID":"b07f9c3c-b503-499b-a1d6-7e8b99698b00","Type":"ContainerStarted","Data":"cfc788fda0b72f1b0b41154a62558fe3f566d1c5de9b62ccb24ff2f7f53793e4"}
Oct 30 13:27:43.369617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:43.369567    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4" podStartSLOduration=1.369550378 podStartE2EDuration="1.369550378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:43.366675238 +0000 UTC m=+3321.422169058" watchObservedRunningTime="2026-10-30 13:27:43.369550378 +0000 UTC m=+3321.425044203"
Oct 30 13:27:54.331824 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:27:54.331790    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:28:12.374857 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:28:12.374819    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:28:37.188969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:37.188880    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"]
Oct 30 13:28:42.190134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.190081    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4" podUID="b07f9c3c-b503-499b-a1d6-7e8b99698b00" containerName="payload-processing" containerID="cri-o://f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741" gracePeriod=25
Oct 30 13:28:42.386758 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:28:42.386621    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:28:42.478438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.478412    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:28:42.517716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.517687    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-routing-overlay-volume\") pod \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") "
Oct 30 13:28:42.517857 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.517829    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-plugins-config-volume\") pod \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") "
Oct 30 13:28:42.517900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.517878    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-kube-api-access-45ktx\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-kube-api-access-45ktx\") pod \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") "
Oct 30 13:28:42.517961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.517949    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-provider-credentials\") pod \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\" (UID: \"b07f9c3c-b503-499b-a1d6-7e8b99698b00\") "
Oct 30 13:28:42.518012 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.517974    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-routing-overlay-volume" pod "b07f9c3c-b503-499b-a1d6-7e8b99698b00" (UID: "b07f9c3c-b503-499b-a1d6-7e8b99698b00"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:42.518243 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.518214    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-plugins-config-volume" pod "b07f9c3c-b503-499b-a1d6-7e8b99698b00" (UID: "b07f9c3c-b503-499b-a1d6-7e8b99698b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:42.518607 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.518591    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:28:42.518658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.518611    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07f9c3c-b503-499b-a1d6-7e8b99698b00-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:28:42.520190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.520169    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-kube-api-access-45ktx" pod "b07f9c3c-b503-499b-a1d6-7e8b99698b00" (UID: "b07f9c3c-b503-499b-a1d6-7e8b99698b00"). InnerVolumeSpecName "kube-api-access-45ktx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:42.520335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.520197    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-provider-credentials" pod "b07f9c3c-b503-499b-a1d6-7e8b99698b00" (UID: "b07f9c3c-b503-499b-a1d6-7e8b99698b00"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:42.619577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.619546    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ktx\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-kube-api-access-45ktx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:28:42.619577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.619576    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07f9c3c-b503-499b-a1d6-7e8b99698b00-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:28:42.808819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.808729    4429 generic.go:356] "Generic (PLEG): container finished" podID="b07f9c3c-b503-499b-a1d6-7e8b99698b00" containerID="f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741" exitCode=0
Oct 30 13:28:42.808819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.808783    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"
Oct 30 13:28:42.808819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.808806    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4" event={"ID":"b07f9c3c-b503-499b-a1d6-7e8b99698b00","Type":"ContainerDied","Data":"f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741"}
Oct 30 13:28:42.809089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.808861    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4" event={"ID":"b07f9c3c-b503-499b-a1d6-7e8b99698b00","Type":"ContainerDied","Data":"cfc788fda0b72f1b0b41154a62558fe3f566d1c5de9b62ccb24ff2f7f53793e4"}
Oct 30 13:28:42.809089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.808888    4429 scope.go:122] "RemoveContainer" containerID="f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741"
Oct 30 13:28:42.823677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.823658    4429 scope.go:122] "RemoveContainer" containerID="f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741"
Oct 30 13:28:42.823989 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:28:42.823969    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741\": container with ID starting with f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741 not found: ID does not exist" containerID="f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741"
Oct 30 13:28:42.824085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.824000    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741"} err="failed to get container status \"f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741\": rpc error: code = NotFound desc = could not find container \"f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741\": container with ID starting with f722a48887aaf02daffe78a9aa8a2a7102147ae0f5be977219975c39e1e22741 not found: ID does not exist"
Oct 30 13:28:42.867269 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.867247    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"]
Oct 30 13:28:42.880874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:42.880855    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-7bb4qzdn4"]
Oct 30 13:28:44.077517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:44.077486    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07f9c3c-b503-499b-a1d6-7e8b99698b00" path="/var/lib/kubelet/pods/b07f9c3c-b503-499b-a1d6-7e8b99698b00/volumes"
Oct 30 13:28:55.742960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.742925    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"]
Oct 30 13:28:55.743983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.743959    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b07f9c3c-b503-499b-a1d6-7e8b99698b00" containerName="payload-processing"
Oct 30 13:28:55.744092 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.743983    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07f9c3c-b503-499b-a1d6-7e8b99698b00" containerName="payload-processing"
Oct 30 13:28:55.744223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.744208    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07f9c3c-b503-499b-a1d6-7e8b99698b00" containerName="payload-processing"
Oct 30 13:28:55.747626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.747605    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:55.766320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.766252    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"]
Oct 30 13:28:55.784248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.775925    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-e2e--8217e17a-dockercfg-zk6fn\""
Oct 30 13:28:55.784248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.776119    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"routing-overlay\""
Oct 30 13:28:55.784248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.776121    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"openshift-service-ca.crt\""
Oct 30 13:28:55.784248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.776207    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1957dc\""
Oct 30 13:28:55.784248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.776360    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"e2e-cfl-6723b8-provider-cred\""
Oct 30 13:28:55.784248 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.778120    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"kube-root-ca.crt\""
Oct 30 13:28:55.930016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.929986    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:55.930202 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.930152    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:55.930266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.930250    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:55.930368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:55.930340    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwb7h\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-kube-api-access-mwb7h\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.031602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.031517    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwb7h\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-kube-api-access-mwb7h\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.031784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.031607    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.031784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.031716    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.031784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.031774    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.032520 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.032493    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.032600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.032584    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.034336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.034317    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.056939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.056911    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwb7h\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-kube-api-access-mwb7h\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.062877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.062854    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.371838 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:28:56.371795    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44164244_aa4e_45a8_86ca_ffa561bd8724.slice/crio-71fb4c1f09ebb0a7a4ef0b351ac1d480ba025b1a13b3825cd9830db443555f4e WatchSource:0}: Error finding container 71fb4c1f09ebb0a7a4ef0b351ac1d480ba025b1a13b3825cd9830db443555f4e: Status 404 returned error can't find the container with id 71fb4c1f09ebb0a7a4ef0b351ac1d480ba025b1a13b3825cd9830db443555f4e
Oct 30 13:28:56.372905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.372858    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"]
Oct 30 13:28:56.926225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.926163    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx" event={"ID":"44164244-aa4e-45a8-86ca-ffa561bd8724","Type":"ContainerStarted","Data":"e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730"}
Oct 30 13:28:56.926891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.926856    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx" event={"ID":"44164244-aa4e-45a8-86ca-ffa561bd8724","Type":"ContainerStarted","Data":"71fb4c1f09ebb0a7a4ef0b351ac1d480ba025b1a13b3825cd9830db443555f4e"}
Oct 30 13:28:56.927395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.927363    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:28:56.976720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:56.976670    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx" podStartSLOduration=1.9766559400000001 podStartE2EDuration="1.97665594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:56.974069983 +0000 UTC m=+3395.029563824" watchObservedRunningTime="2026-10-30 13:28:56.97665594 +0000 UTC m=+3395.032149763"
Oct 30 13:28:58.346310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.335765    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"]
Oct 30 13:28:58.379154 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.379122    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"]
Oct 30 13:28:58.379340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.379247    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.414883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.414847    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-e2e--77992003-dockercfg-zp5z8\""
Oct 30 13:28:58.415083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.414967    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8740d4\""
Oct 30 13:28:58.415556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.415397    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"routing-overlay\""
Oct 30 13:28:58.428692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.428662    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"agc-cred-091b9769f7e9\""
Oct 30 13:28:58.428692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.428676    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"openshift-service-ca.crt\""
Oct 30 13:28:58.428874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.428701    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8740d4\"/\"kube-root-ca.crt\""
Oct 30 13:28:58.482299 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.482248    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.482459 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.482381    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.482531 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.482512    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5xk\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-kube-api-access-km5xk\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.482688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.482611    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.584344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.584311    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5xk\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-kube-api-access-km5xk\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.584496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.584402    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.584540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.584524    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.584580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.584560    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.585210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.585182    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.585497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.585344    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.587214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.587193    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.609451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.609398    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5xk\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-kube-api-access-km5xk\") pod \"payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") " pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:58.729874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:58.729839    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:28:59.035688 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:28:59.035659    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e729d6_5f14_4e8c_9860_7016261b594e.slice/crio-31eecde50e315a6bf03d29555c642094d165fcd5cc4df8ceffab82d7966b7737 WatchSource:0}: Error finding container 31eecde50e315a6bf03d29555c642094d165fcd5cc4df8ceffab82d7966b7737: Status 404 returned error can't find the container with id 31eecde50e315a6bf03d29555c642094d165fcd5cc4df8ceffab82d7966b7737
Oct 30 13:28:59.036691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:28:59.036663    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"]
Oct 30 13:29:00.009704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:00.009660    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7" event={"ID":"d4e729d6-5f14-4e8c-9860-7016261b594e","Type":"ContainerStarted","Data":"f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731"}
Oct 30 13:29:00.010168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:00.009726    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7" event={"ID":"d4e729d6-5f14-4e8c-9860-7016261b594e","Type":"ContainerStarted","Data":"31eecde50e315a6bf03d29555c642094d165fcd5cc4df8ceffab82d7966b7737"}
Oct 30 13:29:00.010168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:00.009757    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:29:00.057902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:00.057858    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7" podStartSLOduration=2.057843554 podStartE2EDuration="2.057843554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:00.056916614 +0000 UTC m=+3398.112410439" watchObservedRunningTime="2026-10-30 13:29:00.057843554 +0000 UTC m=+3398.113337379"
Oct 30 13:29:07.993271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:07.993243    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:29:10.562900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:10.562868    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"]
Oct 30 13:29:10.566369 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:10.566346    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:29:12.423307 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:29:12.418227    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:29:15.564048 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.563993    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7" podUID="d4e729d6-5f14-4e8c-9860-7016261b594e" containerName="payload-processing" containerID="cri-o://f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731" gracePeriod=25
Oct 30 13:29:15.845528 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.845505    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:29:15.853873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.853846    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-plugins-config-volume\") pod \"d4e729d6-5f14-4e8c-9860-7016261b594e\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") "
Oct 30 13:29:15.853998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.853899    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-kube-api-access-km5xk\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-kube-api-access-km5xk\") pod \"d4e729d6-5f14-4e8c-9860-7016261b594e\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") "
Oct 30 13:29:15.853998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.853931    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-provider-credentials\") pod \"d4e729d6-5f14-4e8c-9860-7016261b594e\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") "
Oct 30 13:29:15.854117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.854095    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-routing-overlay-volume\") pod \"d4e729d6-5f14-4e8c-9860-7016261b594e\" (UID: \"d4e729d6-5f14-4e8c-9860-7016261b594e\") "
Oct 30 13:29:15.854323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.854276    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-plugins-config-volume" pod "d4e729d6-5f14-4e8c-9860-7016261b594e" (UID: "d4e729d6-5f14-4e8c-9860-7016261b594e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:15.854427 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.854403    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-routing-overlay-volume" pod "d4e729d6-5f14-4e8c-9860-7016261b594e" (UID: "d4e729d6-5f14-4e8c-9860-7016261b594e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:15.854574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.854556    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.854635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.854579    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e729d6-5f14-4e8c-9860-7016261b594e-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.856226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.856206    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-provider-credentials" pod "d4e729d6-5f14-4e8c-9860-7016261b594e" (UID: "d4e729d6-5f14-4e8c-9860-7016261b594e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:15.856293 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.856236    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-kube-api-access-km5xk" pod "d4e729d6-5f14-4e8c-9860-7016261b594e" (UID: "d4e729d6-5f14-4e8c-9860-7016261b594e"). InnerVolumeSpecName "kube-api-access-km5xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:15.955702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.955671    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km5xk\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-kube-api-access-km5xk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:29:15.955702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:15.955702    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e729d6-5f14-4e8c-9860-7016261b594e-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:29:16.134946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.134916    4429 generic.go:356] "Generic (PLEG): container finished" podID="d4e729d6-5f14-4e8c-9860-7016261b594e" containerID="f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731" exitCode=0
Oct 30 13:29:16.134946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.134948    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7" event={"ID":"d4e729d6-5f14-4e8c-9860-7016261b594e","Type":"ContainerDied","Data":"f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731"}
Oct 30 13:29:16.135173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.134966    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"
Oct 30 13:29:16.135173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.134983    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7" event={"ID":"d4e729d6-5f14-4e8c-9860-7016261b594e","Type":"ContainerDied","Data":"31eecde50e315a6bf03d29555c642094d165fcd5cc4df8ceffab82d7966b7737"}
Oct 30 13:29:16.135173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.135002    4429 scope.go:122] "RemoveContainer" containerID="f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731"
Oct 30 13:29:16.145157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.145139    4429 scope.go:122] "RemoveContainer" containerID="f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731"
Oct 30 13:29:16.145473 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:29:16.145452    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731\": container with ID starting with f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731 not found: ID does not exist" containerID="f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731"
Oct 30 13:29:16.145559 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.145486    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731"} err="failed to get container status \"f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731\": rpc error: code = NotFound desc = could not find container \"f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731\": container with ID starting with f0249d7ab13033b8288253bcee802bf953a7f4500fe4b3b87534aa2c58587731 not found: ID does not exist"
Oct 30 13:29:16.194538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.194515    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"]
Oct 30 13:29:16.202065 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:16.202045    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8740d4/payload-processing-external-model-e2e-extmod-b-8740d4-6886ns4b7"]
Oct 30 13:29:18.076337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:18.076305    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e729d6-5f14-4e8c-9860-7016261b594e" path="/var/lib/kubelet/pods/d4e729d6-5f14-4e8c-9860-7016261b594e/volumes"
Oct 30 13:29:23.360744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:23.360710    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"]
Oct 30 13:29:28.361984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.361940    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s" podUID="0b02ed48-e42c-49d6-9383-f8f73690a947" containerName="payload-processing" containerID="cri-o://4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83" gracePeriod=25
Oct 30 13:29:28.634328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.634237    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:29:28.720793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.720759    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-kube-api-access-qfpx2\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-kube-api-access-qfpx2\") pod \"0b02ed48-e42c-49d6-9383-f8f73690a947\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") "
Oct 30 13:29:28.721009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.720835    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-routing-overlay-volume\") pod \"0b02ed48-e42c-49d6-9383-f8f73690a947\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") "
Oct 30 13:29:28.721009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.720892    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-plugins-config-volume\") pod \"0b02ed48-e42c-49d6-9383-f8f73690a947\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") "
Oct 30 13:29:28.721138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.721013    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-provider-credentials\") pod \"0b02ed48-e42c-49d6-9383-f8f73690a947\" (UID: \"0b02ed48-e42c-49d6-9383-f8f73690a947\") "
Oct 30 13:29:28.721712 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.721487    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-plugins-config-volume" pod "0b02ed48-e42c-49d6-9383-f8f73690a947" (UID: "0b02ed48-e42c-49d6-9383-f8f73690a947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:28.723322 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.722522    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-routing-overlay-volume" pod "0b02ed48-e42c-49d6-9383-f8f73690a947" (UID: "0b02ed48-e42c-49d6-9383-f8f73690a947"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:28.729979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.729956    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-provider-credentials" pod "0b02ed48-e42c-49d6-9383-f8f73690a947" (UID: "0b02ed48-e42c-49d6-9383-f8f73690a947"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:28.730108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.729987    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-kube-api-access-qfpx2" pod "0b02ed48-e42c-49d6-9383-f8f73690a947" (UID: "0b02ed48-e42c-49d6-9383-f8f73690a947"). InnerVolumeSpecName "kube-api-access-qfpx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:28.823183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.823147    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfpx2\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-kube-api-access-qfpx2\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:29:28.823183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.823176    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:29:28.823183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.823186    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b02ed48-e42c-49d6-9383-f8f73690a947-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:29:28.823423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:28.823195    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b02ed48-e42c-49d6-9383-f8f73690a947-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:29:29.263561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.263521    4429 generic.go:356] "Generic (PLEG): container finished" podID="0b02ed48-e42c-49d6-9383-f8f73690a947" containerID="4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83" exitCode=0
Oct 30 13:29:29.263799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.263597    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"
Oct 30 13:29:29.263799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.263620    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s" event={"ID":"0b02ed48-e42c-49d6-9383-f8f73690a947","Type":"ContainerDied","Data":"4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83"}
Oct 30 13:29:29.263799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.263677    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s" event={"ID":"0b02ed48-e42c-49d6-9383-f8f73690a947","Type":"ContainerDied","Data":"1cf4fc2b438c73eb58f668cdd6cc75014c087feec94634d40a1472452811163f"}
Oct 30 13:29:29.263799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.263699    4429 scope.go:122] "RemoveContainer" containerID="4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83"
Oct 30 13:29:29.279756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.279727    4429 scope.go:122] "RemoveContainer" containerID="4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83"
Oct 30 13:29:29.280338 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:29:29.280273    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83\": container with ID starting with 4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83 not found: ID does not exist" containerID="4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83"
Oct 30 13:29:29.280471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.280350    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83"} err="failed to get container status \"4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83\": rpc error: code = NotFound desc = could not find container \"4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83\": container with ID starting with 4572ca735ef80a782504dea61d6d10e883a5f96291ebcdd5686df58247e18b83 not found: ID does not exist"
Oct 30 13:29:29.321909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.321879    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"]
Oct 30 13:29:29.335524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:29.335494    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0505b0/payload-processing-external-model-e2e-extmod-a-0505b0-79742xb6s"]
Oct 30 13:29:30.076393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:30.076361    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b02ed48-e42c-49d6-9383-f8f73690a947" path="/var/lib/kubelet/pods/0b02ed48-e42c-49d6-9383-f8f73690a947/volumes"
Oct 30 13:29:42.372313 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:29:42.372252    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:29:50.617445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.617410    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"]
Oct 30 13:29:50.618233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.618215    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d4e729d6-5f14-4e8c-9860-7016261b594e" containerName="payload-processing"
Oct 30 13:29:50.618296 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.618235    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e729d6-5f14-4e8c-9860-7016261b594e" containerName="payload-processing"
Oct 30 13:29:50.618296 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.618248    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0b02ed48-e42c-49d6-9383-f8f73690a947" containerName="payload-processing"
Oct 30 13:29:50.618296 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.618253    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b02ed48-e42c-49d6-9383-f8f73690a947" containerName="payload-processing"
Oct 30 13:29:50.618407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.618386    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e729d6-5f14-4e8c-9860-7016261b594e" containerName="payload-processing"
Oct 30 13:29:50.618407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.618396    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b02ed48-e42c-49d6-9383-f8f73690a947" containerName="payload-processing"
Oct 30 13:29:50.623644 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.623623    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.641155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.641130    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"]
Oct 30 13:29:50.650351 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.650330    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 13:29:50.650488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.650447    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 13:29:50.650547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.650483    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 13:29:50.650547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.650496    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-g9xns\""
Oct 30 13:29:50.654390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.654363    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-plugins-config-volume\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.654541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.654404    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9sgw\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-kube-api-access-s9sgw\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.654619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.654567    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-provider-credentials\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.654727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.654708    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-routing-overlay-volume\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.655340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.655320    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-46a10b0d3d71\""
Oct 30 13:29:50.655438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.655391    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 13:29:50.756261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.756230    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-plugins-config-volume\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.756261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.756263    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9sgw\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-kube-api-access-s9sgw\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.756503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.756339    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-provider-credentials\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.756503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.756409    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-routing-overlay-volume\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.756983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.756951    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-routing-overlay-volume\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.757117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.757033    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-plugins-config-volume\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.758963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.758944    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-provider-credentials\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.785524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.785485    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9sgw\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-kube-api-access-s9sgw\") pod \"payload-processing-external-model-7f64b5b66-lh4pd\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") " pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:50.939121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:50.939090    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:51.255450 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:29:51.255422    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2aef14_1e8a_48fd_9ef5_39a49aa7e0fe.slice/crio-362786a7ae13c1d545f7371b3890fe711b635c1fdf9d6f507755a17c02c49a46 WatchSource:0}: Error finding container 362786a7ae13c1d545f7371b3890fe711b635c1fdf9d6f507755a17c02c49a46: Status 404 returned error can't find the container with id 362786a7ae13c1d545f7371b3890fe711b635c1fdf9d6f507755a17c02c49a46
Oct 30 13:29:51.256876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:51.256849    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"]
Oct 30 13:29:51.514043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:51.513952    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd" event={"ID":"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe","Type":"ContainerStarted","Data":"9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a"}
Oct 30 13:29:51.514043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:51.514004    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd" event={"ID":"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe","Type":"ContainerStarted","Data":"362786a7ae13c1d545f7371b3890fe711b635c1fdf9d6f507755a17c02c49a46"}
Oct 30 13:29:51.514316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:51.514183    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:29:51.562986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:51.562941    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd" podStartSLOduration=1.562929364 podStartE2EDuration="1.562929364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:51.561364973 +0000 UTC m=+3449.616858816" watchObservedRunningTime="2026-10-30 13:29:51.562929364 +0000 UTC m=+3449.618423188"
Oct 30 13:29:55.251458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:29:55.251420    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"]
Oct 30 13:30:00.252561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.252508    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd" podUID="8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" containerName="payload-processing" containerID="cri-o://9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a" gracePeriod=25
Oct 30 13:30:00.377991 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.377960    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"]
Oct 30 13:30:00.384267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.384244    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:00.425644 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.425601    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"]
Oct 30 13:30:00.440025 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.431877    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"]
Oct 30 13:30:00.440025 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.439463    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:00.452701 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.452141    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"]
Oct 30 13:30:00.458940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.458922    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:00.473329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.473305    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"]
Oct 30 13:30:00.473887 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.473870    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:30:00.507572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.507493    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"]
Oct 30 13:30:00.521228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.521190    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"]
Oct 30 13:30:00.521412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.521351    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:00.545262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.545233    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"]
Oct 30 13:30:00.551899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.551753    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j\" (UID: \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:00.552057 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.551949    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a81ea6-9007-4f05-9780-deaab8f77dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn\" (UID: \"d5a81ea6-9007-4f05-9780-deaab8f77dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:00.552057 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.552018    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7tql\" (UniqueName: \"kubernetes.io/projected/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-kube-api-access-d7tql\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j\" (UID: \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:00.552186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.552118    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngsc\" (UniqueName: \"kubernetes.io/projected/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-kube-api-access-pngsc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88\" (UID: \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:00.552476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.552362    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88\" (UID: \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:00.552476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.552392    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxd9p\" (UniqueName: \"kubernetes.io/projected/d5a81ea6-9007-4f05-9780-deaab8f77dde-kube-api-access-lxd9p\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn\" (UID: \"d5a81ea6-9007-4f05-9780-deaab8f77dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:00.599365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.599342    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:30:00.599502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.599372    4429 generic.go:356] "Generic (PLEG): container finished" podID="8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" containerID="9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a" exitCode=0
Oct 30 13:30:00.599502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.599439    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd" event={"ID":"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe","Type":"ContainerDied","Data":"9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a"}
Oct 30 13:30:00.599502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.599493    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd" event={"ID":"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe","Type":"ContainerDied","Data":"362786a7ae13c1d545f7371b3890fe711b635c1fdf9d6f507755a17c02c49a46"}
Oct 30 13:30:00.599614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.599524    4429 scope.go:122] "RemoveContainer" containerID="9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a"
Oct 30 13:30:00.609613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.609596    4429 scope.go:122] "RemoveContainer" containerID="9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a"
Oct 30 13:30:00.609909 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:00.609890    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a\": container with ID starting with 9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a not found: ID does not exist" containerID="9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a"
Oct 30 13:30:00.609974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.609918    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a"} err="failed to get container status \"9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a\": rpc error: code = NotFound desc = could not find container \"9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a\": container with ID starting with 9775d97a1b5a3da52ec2ceade7853771ce8656cef288b7f0be85298af77d182a not found: ID does not exist"
Oct 30 13:30:00.654656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.654633    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j\" (UID: \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:00.654824 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.654673    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a81ea6-9007-4f05-9780-deaab8f77dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn\" (UID: \"d5a81ea6-9007-4f05-9780-deaab8f77dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:00.654824 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.654714    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7tql\" (UniqueName: \"kubernetes.io/projected/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-kube-api-access-d7tql\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j\" (UID: \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:00.654824 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.654744    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xp7f\" (UniqueName: \"kubernetes.io/projected/b0f8d184-a27d-4df3-a63b-42008999091b-kube-api-access-4xp7f\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx\" (UID: \"b0f8d184-a27d-4df3-a63b-42008999091b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:00.655001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.654834    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngsc\" (UniqueName: \"kubernetes.io/projected/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-kube-api-access-pngsc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88\" (UID: \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:00.655066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.655001    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f8d184-a27d-4df3-a63b-42008999091b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx\" (UID: \"b0f8d184-a27d-4df3-a63b-42008999091b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:00.655227 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.655200    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88\" (UID: \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:00.655314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.655244    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxd9p\" (UniqueName: \"kubernetes.io/projected/d5a81ea6-9007-4f05-9780-deaab8f77dde-kube-api-access-lxd9p\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn\" (UID: \"d5a81ea6-9007-4f05-9780-deaab8f77dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:00.655386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.655367    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a81ea6-9007-4f05-9780-deaab8f77dde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn\" (UID: \"d5a81ea6-9007-4f05-9780-deaab8f77dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:00.655386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.655376    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j\" (UID: \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:00.655816 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.655801    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88\" (UID: \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:00.723317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.723265    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxd9p\" (UniqueName: \"kubernetes.io/projected/d5a81ea6-9007-4f05-9780-deaab8f77dde-kube-api-access-lxd9p\") pod \"maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn\" (UID: \"d5a81ea6-9007-4f05-9780-deaab8f77dde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:00.723469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.723328    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngsc\" (UniqueName: \"kubernetes.io/projected/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-kube-api-access-pngsc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88\" (UID: \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:00.746139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.746114    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7tql\" (UniqueName: \"kubernetes.io/projected/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-kube-api-access-d7tql\") pod \"maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j\" (UID: \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:00.756052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.756031    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-kube-api-access-s9sgw\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-kube-api-access-s9sgw\") pod \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") "
Oct 30 13:30:00.756173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.756148    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-plugins-config-volume\") pod \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") "
Oct 30 13:30:00.756228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.756189    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-provider-credentials\") pod \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") "
Oct 30 13:30:00.756410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.756395    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-routing-overlay-volume\") pod \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\" (UID: \"8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe\") "
Oct 30 13:30:00.756592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.756565    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-plugins-config-volume" pod "8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" (UID: "8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:00.756760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.756742    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-routing-overlay-volume" pod "8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" (UID: "8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:00.756806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.756759    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f8d184-a27d-4df3-a63b-42008999091b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx\" (UID: \"b0f8d184-a27d-4df3-a63b-42008999091b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:00.756951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.756936    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xp7f\" (UniqueName: \"kubernetes.io/projected/b0f8d184-a27d-4df3-a63b-42008999091b-kube-api-access-4xp7f\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx\" (UID: \"b0f8d184-a27d-4df3-a63b-42008999091b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:00.757031 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.757021    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:00.757095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.757035    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:00.757430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.757411    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f8d184-a27d-4df3-a63b-42008999091b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx\" (UID: \"b0f8d184-a27d-4df3-a63b-42008999091b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:00.758249 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.758213    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-kube-api-access-s9sgw" pod "8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" (UID: "8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe"). InnerVolumeSpecName "kube-api-access-s9sgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:00.758465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.758446    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-provider-credentials" pod "8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" (UID: "8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:00.760404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.760389    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:00.781357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.781336    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:00.813037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.812980    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xp7f\" (UniqueName: \"kubernetes.io/projected/b0f8d184-a27d-4df3-a63b-42008999091b-kube-api-access-4xp7f\") pod \"maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx\" (UID: \"b0f8d184-a27d-4df3-a63b-42008999091b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:00.843237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.842270    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:00.859224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.859192    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9sgw\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-kube-api-access-s9sgw\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:00.859402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:00.859229    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:01.008577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.008505    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:01.098126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.098095    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"]
Oct 30 13:30:01.124774 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:30:01.124596    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c2e945_b9c1_40a4_9f2f_e411b9335c46.slice/crio-ff201d9180c5d7acb5a75b0a0203b2d8b21a3fa4bdfa440a2cfff143c0d057e8 WatchSource:0}: Error finding container ff201d9180c5d7acb5a75b0a0203b2d8b21a3fa4bdfa440a2cfff143c0d057e8: Status 404 returned error can't find the container with id ff201d9180c5d7acb5a75b0a0203b2d8b21a3fa4bdfa440a2cfff143c0d057e8
Oct 30 13:30:01.126249 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.126220    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"]
Oct 30 13:30:01.185259 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:30:01.185233    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f8d184_a27d_4df3_a63b_42008999091b.slice/crio-3e1baeb9df55148cc09060dd5f80856f4e3f28f524610eae310a348d9ee6fdd7 WatchSource:0}: Error finding container 3e1baeb9df55148cc09060dd5f80856f4e3f28f524610eae310a348d9ee6fdd7: Status 404 returned error can't find the container with id 3e1baeb9df55148cc09060dd5f80856f4e3f28f524610eae310a348d9ee6fdd7
Oct 30 13:30:01.185621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.185598    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"]
Oct 30 13:30:01.331196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.331167    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"]
Oct 30 13:30:01.352267 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:30:01.352239    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd058e91_9991_4df1_91fa_1e2a6e5fbcb9.slice/crio-051cb1a618bb73100f66e149e148a613c3152396eb67b12b9af1142121601845 WatchSource:0}: Error finding container 051cb1a618bb73100f66e149e148a613c3152396eb67b12b9af1142121601845: Status 404 returned error can't find the container with id 051cb1a618bb73100f66e149e148a613c3152396eb67b12b9af1142121601845
Oct 30 13:30:01.607757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.607682    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" event={"ID":"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9","Type":"ContainerStarted","Data":"051cb1a618bb73100f66e149e148a613c3152396eb67b12b9af1142121601845"}
Oct 30 13:30:01.609581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.609553    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" event={"ID":"d0c2e945-b9c1-40a4-9f2f-e411b9335c46","Type":"ContainerStarted","Data":"294e8128a3a13df84dbb8cc7874c0cfefa5c0477d59f31258f9a3620ca297bc5"}
Oct 30 13:30:01.609716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.609613    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" event={"ID":"d0c2e945-b9c1-40a4-9f2f-e411b9335c46","Type":"ContainerStarted","Data":"ff201d9180c5d7acb5a75b0a0203b2d8b21a3fa4bdfa440a2cfff143c0d057e8"}
Oct 30 13:30:01.610797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.610774    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" event={"ID":"b0f8d184-a27d-4df3-a63b-42008999091b","Type":"ContainerStarted","Data":"3e1baeb9df55148cc09060dd5f80856f4e3f28f524610eae310a348d9ee6fdd7"}
Oct 30 13:30:01.611695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.611673    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"
Oct 30 13:30:01.613441 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.613415    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" event={"ID":"d5a81ea6-9007-4f05-9780-deaab8f77dde","Type":"ContainerStarted","Data":"897c962d00cf476148708fb4f1628262eee778529695e020e62bef1397a93f16"}
Oct 30 13:30:01.613524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.613462    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" event={"ID":"d5a81ea6-9007-4f05-9780-deaab8f77dde","Type":"ContainerStarted","Data":"436fb5bf95217ab3decaa9db1927821353ee304f17a8f0b347a6caa3192852d6"}
Oct 30 13:30:01.657873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.657833    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" podStartSLOduration=1.6578186719999999 podStartE2EDuration="1.657818672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:01.654231651 +0000 UTC m=+3459.709725504" watchObservedRunningTime="2026-10-30 13:30:01.657818672 +0000 UTC m=+3459.713312496"
Oct 30 13:30:01.697342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.697300    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"]
Oct 30 13:30:01.713897 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.713875    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f64b5b66-lh4pd"]
Oct 30 13:30:01.763503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:01.763459    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" podStartSLOduration=1.763446198 podStartE2EDuration="1.763446198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:01.760684243 +0000 UTC m=+3459.816178068" watchObservedRunningTime="2026-10-30 13:30:01.763446198 +0000 UTC m=+3459.818940021"
Oct 30 13:30:02.079252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.079223    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" path="/var/lib/kubelet/pods/8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe/volumes"
Oct 30 13:30:02.622560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.622486    4429 generic.go:356] "Generic (PLEG): container finished" podID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerID="6d23fce636d83b3cff08561915a477f26b1b778538e4f3e3a7869f253de01ffa" exitCode=60
Oct 30 13:30:02.622985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.622575    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" event={"ID":"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9","Type":"ContainerDied","Data":"6d23fce636d83b3cff08561915a477f26b1b778538e4f3e3a7869f253de01ffa"}
Oct 30 13:30:02.622985 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.622853    4429 scope.go:122] "RemoveContainer" containerID="6d23fce636d83b3cff08561915a477f26b1b778538e4f3e3a7869f253de01ffa"
Oct 30 13:30:02.624465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.624441    4429 generic.go:356] "Generic (PLEG): container finished" podID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerID="294e8128a3a13df84dbb8cc7874c0cfefa5c0477d59f31258f9a3620ca297bc5" exitCode=60
Oct 30 13:30:02.624601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.624525    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" event={"ID":"d0c2e945-b9c1-40a4-9f2f-e411b9335c46","Type":"ContainerDied","Data":"294e8128a3a13df84dbb8cc7874c0cfefa5c0477d59f31258f9a3620ca297bc5"}
Oct 30 13:30:02.624754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.624739    4429 scope.go:122] "RemoveContainer" containerID="294e8128a3a13df84dbb8cc7874c0cfefa5c0477d59f31258f9a3620ca297bc5"
Oct 30 13:30:02.626497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.626472    4429 generic.go:356] "Generic (PLEG): container finished" podID="b0f8d184-a27d-4df3-a63b-42008999091b" containerID="e61a8526c4326b7c70b5dc458069adfd995314a69b9fd7eaa12de533ad06af65" exitCode=60
Oct 30 13:30:02.626610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.626579    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" event={"ID":"b0f8d184-a27d-4df3-a63b-42008999091b","Type":"ContainerDied","Data":"e61a8526c4326b7c70b5dc458069adfd995314a69b9fd7eaa12de533ad06af65"}
Oct 30 13:30:02.626860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.626842    4429 scope.go:122] "RemoveContainer" containerID="e61a8526c4326b7c70b5dc458069adfd995314a69b9fd7eaa12de533ad06af65"
Oct 30 13:30:02.628958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.628938    4429 generic.go:356] "Generic (PLEG): container finished" podID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerID="897c962d00cf476148708fb4f1628262eee778529695e020e62bef1397a93f16" exitCode=60
Oct 30 13:30:02.629061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.628987    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" event={"ID":"d5a81ea6-9007-4f05-9780-deaab8f77dde","Type":"ContainerDied","Data":"897c962d00cf476148708fb4f1628262eee778529695e020e62bef1397a93f16"}
Oct 30 13:30:02.629193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:02.629178    4429 scope.go:122] "RemoveContainer" containerID="897c962d00cf476148708fb4f1628262eee778529695e020e62bef1397a93f16"
Oct 30 13:30:03.638781 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.638744    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" event={"ID":"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9","Type":"ContainerStarted","Data":"44bf4723cc5dd30f356e9783284ecffe6341dbcb123ad0ab830aa1356dcaf45c"}
Oct 30 13:30:03.640892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.640837    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" event={"ID":"d0c2e945-b9c1-40a4-9f2f-e411b9335c46","Type":"ContainerStarted","Data":"1e277dda0dfa7c533888113714dabace7e87e690d2120cca0021c596e53f5e64"}
Oct 30 13:30:03.642668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.642646    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" event={"ID":"b0f8d184-a27d-4df3-a63b-42008999091b","Type":"ContainerStarted","Data":"6687c2ef1cad026b556d8b611abe42add8a14a1f952bf87e5813af5af0696af6"}
Oct 30 13:30:03.644569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.644547    4429 generic.go:356] "Generic (PLEG): container finished" podID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerID="ea4dfd6a9dc3217ed31640798b5ea9aa39391e1f55b0b6accf6ddad90f8aec0f" exitCode=60
Oct 30 13:30:03.644700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.644628    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" event={"ID":"d5a81ea6-9007-4f05-9780-deaab8f77dde","Type":"ContainerDied","Data":"ea4dfd6a9dc3217ed31640798b5ea9aa39391e1f55b0b6accf6ddad90f8aec0f"}
Oct 30 13:30:03.644700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.644687    4429 scope.go:122] "RemoveContainer" containerID="897c962d00cf476148708fb4f1628262eee778529695e020e62bef1397a93f16"
Oct 30 13:30:03.644842 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.644827    4429 scope.go:122] "RemoveContainer" containerID="ea4dfd6a9dc3217ed31640798b5ea9aa39391e1f55b0b6accf6ddad90f8aec0f"
Oct 30 13:30:03.645115 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:03.645096    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn_odh-ai-gateway-infra(d5a81ea6-9007-4f05-9780-deaab8f77dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde"
Oct 30 13:30:03.707395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.704246    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" podStartSLOduration=3.704228987 podStartE2EDuration="3.704228987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.700187136 +0000 UTC m=+3461.755680959" watchObservedRunningTime="2026-10-30 13:30:03.704228987 +0000 UTC m=+3461.759722812"
Oct 30 13:30:03.928109 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:03.928053    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" podStartSLOduration=3.928038845 podStartE2EDuration="3.928038845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.924057034 +0000 UTC m=+3461.979550860" watchObservedRunningTime="2026-10-30 13:30:03.928038845 +0000 UTC m=+3461.983532669"
Oct 30 13:30:04.653741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.653694    4429 scope.go:122] "RemoveContainer" containerID="ea4dfd6a9dc3217ed31640798b5ea9aa39391e1f55b0b6accf6ddad90f8aec0f"
Oct 30 13:30:04.654203 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:04.654073    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn_odh-ai-gateway-infra(d5a81ea6-9007-4f05-9780-deaab8f77dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde"
Oct 30 13:30:04.655493 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.655467    4429 generic.go:356] "Generic (PLEG): container finished" podID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerID="44bf4723cc5dd30f356e9783284ecffe6341dbcb123ad0ab830aa1356dcaf45c" exitCode=60
Oct 30 13:30:04.655622 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.655518    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" event={"ID":"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9","Type":"ContainerDied","Data":"44bf4723cc5dd30f356e9783284ecffe6341dbcb123ad0ab830aa1356dcaf45c"}
Oct 30 13:30:04.655622 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.655584    4429 scope.go:122] "RemoveContainer" containerID="6d23fce636d83b3cff08561915a477f26b1b778538e4f3e3a7869f253de01ffa"
Oct 30 13:30:04.655906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.655887    4429 scope.go:122] "RemoveContainer" containerID="44bf4723cc5dd30f356e9783284ecffe6341dbcb123ad0ab830aa1356dcaf45c"
Oct 30 13:30:04.656248 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:04.656218    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88_odh-ai-gateway-infra(dd058e91-9991-4df1-91fa-1e2a6e5fbcb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9"
Oct 30 13:30:04.657785 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.657764    4429 generic.go:356] "Generic (PLEG): container finished" podID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerID="1e277dda0dfa7c533888113714dabace7e87e690d2120cca0021c596e53f5e64" exitCode=60
Oct 30 13:30:04.657898 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.657847    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" event={"ID":"d0c2e945-b9c1-40a4-9f2f-e411b9335c46","Type":"ContainerDied","Data":"1e277dda0dfa7c533888113714dabace7e87e690d2120cca0021c596e53f5e64"}
Oct 30 13:30:04.658080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.658066    4429 scope.go:122] "RemoveContainer" containerID="1e277dda0dfa7c533888113714dabace7e87e690d2120cca0021c596e53f5e64"
Oct 30 13:30:04.658404 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:04.658384    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j_odh-ai-gateway-infra(d0c2e945-b9c1-40a4-9f2f-e411b9335c46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46"
Oct 30 13:30:04.659716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.659695    4429 generic.go:356] "Generic (PLEG): container finished" podID="b0f8d184-a27d-4df3-a63b-42008999091b" containerID="6687c2ef1cad026b556d8b611abe42add8a14a1f952bf87e5813af5af0696af6" exitCode=60
Oct 30 13:30:04.659818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.659724    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" event={"ID":"b0f8d184-a27d-4df3-a63b-42008999091b","Type":"ContainerDied","Data":"6687c2ef1cad026b556d8b611abe42add8a14a1f952bf87e5813af5af0696af6"}
Oct 30 13:30:04.659994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.659978    4429 scope.go:122] "RemoveContainer" containerID="6687c2ef1cad026b556d8b611abe42add8a14a1f952bf87e5813af5af0696af6"
Oct 30 13:30:04.660241 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:04.660223    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx_odh-ai-gateway-infra(b0f8d184-a27d-4df3-a63b-42008999091b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" podUID="b0f8d184-a27d-4df3-a63b-42008999091b"
Oct 30 13:30:04.670103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.670083    4429 scope.go:122] "RemoveContainer" containerID="294e8128a3a13df84dbb8cc7874c0cfefa5c0477d59f31258f9a3620ca297bc5"
Oct 30 13:30:04.685026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:04.685001    4429 scope.go:122] "RemoveContainer" containerID="e61a8526c4326b7c70b5dc458069adfd995314a69b9fd7eaa12de533ad06af65"
Oct 30 13:30:05.669811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:05.669782    4429 scope.go:122] "RemoveContainer" containerID="44bf4723cc5dd30f356e9783284ecffe6341dbcb123ad0ab830aa1356dcaf45c"
Oct 30 13:30:05.670256 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:05.670142    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88_odh-ai-gateway-infra(dd058e91-9991-4df1-91fa-1e2a6e5fbcb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9"
Oct 30 13:30:05.671905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:05.671884    4429 scope.go:122] "RemoveContainer" containerID="1e277dda0dfa7c533888113714dabace7e87e690d2120cca0021c596e53f5e64"
Oct 30 13:30:05.672214 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:05.672189    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j_odh-ai-gateway-infra(d0c2e945-b9c1-40a4-9f2f-e411b9335c46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46"
Oct 30 13:30:05.679721 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:05.679700    4429 scope.go:122] "RemoveContainer" containerID="6687c2ef1cad026b556d8b611abe42add8a14a1f952bf87e5813af5af0696af6"
Oct 30 13:30:05.680010 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:05.679988    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx_odh-ai-gateway-infra(b0f8d184-a27d-4df3-a63b-42008999091b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" podUID="b0f8d184-a27d-4df3-a63b-42008999091b"
Oct 30 13:30:07.817225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.817193    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"]
Oct 30 13:30:07.818363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.818342    4429 cpu_manager.go:414] "containerMap: removing container" podUID="8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" containerName="payload-processing"
Oct 30 13:30:07.818491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.818364    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" containerName="payload-processing"
Oct 30 13:30:07.818554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.818543    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2aef14-1e8a-48fd-9ef5-39a49aa7e0fe" containerName="payload-processing"
Oct 30 13:30:07.823872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.823851    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.859551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.859516    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"agc-cred-875e482463fe\""
Oct 30 13:30:07.859751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.859600    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"openshift-service-ca.crt\""
Oct 30 13:30:07.859751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.859627    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"routing-overlay\""
Oct 30 13:30:07.859751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.859688    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"kube-root-ca.crt\""
Oct 30 13:30:07.859984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.859769    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"payload-processing-external-model-e2e--e38313e4-dockercfg-ldjx5\""
Oct 30 13:30:07.859984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.859805    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"payload-processing-external-model-plugins-e2e-extmod-b-75d385\""
Oct 30 13:30:07.861813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.861783    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"]
Oct 30 13:30:07.870914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.870882    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvtd\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-kube-api-access-pwvtd\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.871048 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.871029    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.871304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.871269    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.871405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.871347    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.972884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.972838    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvtd\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-kube-api-access-pwvtd\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.973065 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.972924    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.973160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.973099    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.973160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.973125    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.973994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.973962    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.974151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.974126    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:07.977022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:07.976995    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:08.015731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:08.015429    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvtd\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-kube-api-access-pwvtd\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:08.142562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:08.142489    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:08.452940 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:30:08.452911    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89106ea_620c_4241_a90c_c449c4d866fb.slice/crio-0a60130ba78e1be7a07987c57cf2734bff7c0feefd845c18ecc0b81ab3306a62 WatchSource:0}: Error finding container 0a60130ba78e1be7a07987c57cf2734bff7c0feefd845c18ecc0b81ab3306a62: Status 404 returned error can't find the container with id 0a60130ba78e1be7a07987c57cf2734bff7c0feefd845c18ecc0b81ab3306a62
Oct 30 13:30:08.454533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:08.454513    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"]
Oct 30 13:30:08.714238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:08.714143    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q" event={"ID":"d89106ea-620c-4241-a90c-c449c4d866fb","Type":"ContainerStarted","Data":"33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55"}
Oct 30 13:30:08.714238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:08.714230    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:08.714497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:08.714249    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q" event={"ID":"d89106ea-620c-4241-a90c-c449c4d866fb","Type":"ContainerStarted","Data":"0a60130ba78e1be7a07987c57cf2734bff7c0feefd845c18ecc0b81ab3306a62"}
Oct 30 13:30:08.782628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:08.782566    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q" podStartSLOduration=1.782547104 podStartE2EDuration="1.782547104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:08.779329824 +0000 UTC m=+3466.834823649" watchObservedRunningTime="2026-10-30 13:30:08.782547104 +0000 UTC m=+3466.838040929"
Oct 30 13:30:12.372727 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:12.372690    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:30:13.011338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.011272    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"]
Oct 30 13:30:13.019436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.019397    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.038572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.038546    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"]
Oct 30 13:30:13.170905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.170872    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.171134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.170937    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7rcq\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-kube-api-access-q7rcq\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.171134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.171045    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.171134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.171092    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.171269 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.171239    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.171354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.171332    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.171398 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.171355    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.171398 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.171375    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582c6289-e704-4dbb-869d-42e329231c15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.171488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.171452    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582c6289-e704-4dbb-869d-42e329231c15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.272586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272493    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.272586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272539    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.272586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272568    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582c6289-e704-4dbb-869d-42e329231c15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.272943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272609    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582c6289-e704-4dbb-869d-42e329231c15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.272943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272693    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.272943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272768    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7rcq\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-kube-api-access-q7rcq\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.272943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272840    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.272943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272919    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.273216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.272995    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.273216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.273143    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.273397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.273263    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.273460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.273443    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.273522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.273501    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.273750 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.273729    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582c6289-e704-4dbb-869d-42e329231c15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.275442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.275424    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582c6289-e704-4dbb-869d-42e329231c15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.275627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.275609    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.299949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.299914    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.300095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.299923    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7rcq\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-kube-api-access-q7rcq\") pod \"maas-default-gateway-openshift-default-57f78bcd55-jhgnx\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.340741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.340708    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:13.665153 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:30:13.665125    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582c6289_e704_4dbb_869d_42e329231c15.slice/crio-5312314b972232fd1594ff31d935c0a207d28a8ad14818b66bba5051ec9ad123 WatchSource:0}: Error finding container 5312314b972232fd1594ff31d935c0a207d28a8ad14818b66bba5051ec9ad123: Status 404 returned error can't find the container with id 5312314b972232fd1594ff31d935c0a207d28a8ad14818b66bba5051ec9ad123
Oct 30 13:30:13.666465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.666436    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"]
Oct 30 13:30:13.668547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.668515    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:13.668651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.668583    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:13.668651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.668612    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:13.759074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.759031    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" event={"ID":"582c6289-e704-4dbb-869d-42e329231c15","Type":"ContainerStarted","Data":"b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3"}
Oct 30 13:30:13.759226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.759095    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" event={"ID":"582c6289-e704-4dbb-869d-42e329231c15","Type":"ContainerStarted","Data":"5312314b972232fd1594ff31d935c0a207d28a8ad14818b66bba5051ec9ad123"}
Oct 30 13:30:13.813795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:13.813737    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podStartSLOduration=1.8137178889999999 podStartE2EDuration="1.813717889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:13.809891873 +0000 UTC m=+3471.865385698" watchObservedRunningTime="2026-10-30 13:30:13.813717889 +0000 UTC m=+3471.869211714"
Oct 30 13:30:14.068496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:14.068453    4429 scope.go:122] "RemoveContainer" containerID="1e277dda0dfa7c533888113714dabace7e87e690d2120cca0021c596e53f5e64"
Oct 30 13:30:14.068715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:14.068658    4429 scope.go:122] "RemoveContainer" containerID="ea4dfd6a9dc3217ed31640798b5ea9aa39391e1f55b0b6accf6ddad90f8aec0f"
Oct 30 13:30:14.341688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:14.341601    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:14.343527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:14.343498    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-jhgnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 13:30:14.343661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:14.343548    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 13:30:14.781769 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:14.781738    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" event={"ID":"d0c2e945-b9c1-40a4-9f2f-e411b9335c46","Type":"ContainerStarted","Data":"99589b11f6cdc0575b4b45d281115545e002bd01714d7323c2a978b12f24bcd9"}
Oct 30 13:30:14.783712 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:14.783686    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" event={"ID":"d5a81ea6-9007-4f05-9780-deaab8f77dde","Type":"ContainerStarted","Data":"7eb328557203220c87e962c98f7eab93bbfea2f3458edc0cb806b0705636dbc7"}
Oct 30 13:30:15.341771 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.341739    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-jhgnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 13:30:15.341939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.341811    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 13:30:15.792540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.792502    4429 generic.go:356] "Generic (PLEG): container finished" podID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerID="99589b11f6cdc0575b4b45d281115545e002bd01714d7323c2a978b12f24bcd9" exitCode=60
Oct 30 13:30:15.793010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.792571    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" event={"ID":"d0c2e945-b9c1-40a4-9f2f-e411b9335c46","Type":"ContainerDied","Data":"99589b11f6cdc0575b4b45d281115545e002bd01714d7323c2a978b12f24bcd9"}
Oct 30 13:30:15.793010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.792633    4429 scope.go:122] "RemoveContainer" containerID="1e277dda0dfa7c533888113714dabace7e87e690d2120cca0021c596e53f5e64"
Oct 30 13:30:15.793010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.793000    4429 scope.go:122] "RemoveContainer" containerID="99589b11f6cdc0575b4b45d281115545e002bd01714d7323c2a978b12f24bcd9"
Oct 30 13:30:15.793408 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:15.793382    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j_odh-ai-gateway-infra(d0c2e945-b9c1-40a4-9f2f-e411b9335c46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46"
Oct 30 13:30:15.795074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.795052    4429 generic.go:356] "Generic (PLEG): container finished" podID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerID="7eb328557203220c87e962c98f7eab93bbfea2f3458edc0cb806b0705636dbc7" exitCode=60
Oct 30 13:30:15.795179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.795124    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" event={"ID":"d5a81ea6-9007-4f05-9780-deaab8f77dde","Type":"ContainerDied","Data":"7eb328557203220c87e962c98f7eab93bbfea2f3458edc0cb806b0705636dbc7"}
Oct 30 13:30:15.795526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.795510    4429 scope.go:122] "RemoveContainer" containerID="7eb328557203220c87e962c98f7eab93bbfea2f3458edc0cb806b0705636dbc7"
Oct 30 13:30:15.795806 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:15.795787    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn_odh-ai-gateway-infra(d5a81ea6-9007-4f05-9780-deaab8f77dde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde"
Oct 30 13:30:15.806906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.806888    4429 scope.go:122] "RemoveContainer" containerID="ea4dfd6a9dc3217ed31640798b5ea9aa39391e1f55b0b6accf6ddad90f8aec0f"
Oct 30 13:30:15.841422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.841388    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"]
Oct 30 13:30:15.896368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:15.896337    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"]
Oct 30 13:30:16.073498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:16.072892    4429 scope.go:122] "RemoveContainer" containerID="44bf4723cc5dd30f356e9783284ecffe6341dbcb123ad0ab830aa1356dcaf45c"
Oct 30 13:30:16.341961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:16.341873    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-jhgnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 13:30:16.341961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:16.341931    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 13:30:16.804266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:16.804231    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" event={"ID":"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9","Type":"ContainerStarted","Data":"20ffb7d4b43e46f749051ae8dc6b27f6878cab1dc55e497fbc5cf09b706eea23"}
Oct 30 13:30:17.081359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.077777    4429 scope.go:122] "RemoveContainer" containerID="6687c2ef1cad026b556d8b611abe42add8a14a1f952bf87e5813af5af0696af6"
Oct 30 13:30:17.091581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.091557    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:17.184727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.184687    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:17.233359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.233322    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a81ea6-9007-4f05-9780-deaab8f77dde-kube-api-access-lxd9p\" (UniqueName: \"kubernetes.io/projected/d5a81ea6-9007-4f05-9780-deaab8f77dde-kube-api-access-lxd9p\") pod \"d5a81ea6-9007-4f05-9780-deaab8f77dde\" (UID: \"d5a81ea6-9007-4f05-9780-deaab8f77dde\") "
Oct 30 13:30:17.233578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.233457    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a81ea6-9007-4f05-9780-deaab8f77dde-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a81ea6-9007-4f05-9780-deaab8f77dde-maas-api-service-ca\") pod \"d5a81ea6-9007-4f05-9780-deaab8f77dde\" (UID: \"d5a81ea6-9007-4f05-9780-deaab8f77dde\") "
Oct 30 13:30:17.235320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.235274    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a81ea6-9007-4f05-9780-deaab8f77dde-maas-api-service-ca" pod "d5a81ea6-9007-4f05-9780-deaab8f77dde" (UID: "d5a81ea6-9007-4f05-9780-deaab8f77dde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:17.239508 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.239483    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a81ea6-9007-4f05-9780-deaab8f77dde-kube-api-access-lxd9p" pod "d5a81ea6-9007-4f05-9780-deaab8f77dde" (UID: "d5a81ea6-9007-4f05-9780-deaab8f77dde"). InnerVolumeSpecName "kube-api-access-lxd9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:17.335474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.335403    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-maas-api-service-ca\") pod \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\" (UID: \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\") "
Oct 30 13:30:17.335625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.335486    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-kube-api-access-d7tql\" (UniqueName: \"kubernetes.io/projected/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-kube-api-access-d7tql\") pod \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\" (UID: \"d0c2e945-b9c1-40a4-9f2f-e411b9335c46\") "
Oct 30 13:30:17.336155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.336074    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxd9p\" (UniqueName: \"kubernetes.io/projected/d5a81ea6-9007-4f05-9780-deaab8f77dde-kube-api-access-lxd9p\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.336155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.336099    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a81ea6-9007-4f05-9780-deaab8f77dde-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.336532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.336488    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-maas-api-service-ca" pod "d0c2e945-b9c1-40a4-9f2f-e411b9335c46" (UID: "d0c2e945-b9c1-40a4-9f2f-e411b9335c46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:17.342063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.341927    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-jhgnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 13:30:17.342063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.341976    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 13:30:17.344731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.344707    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-kube-api-access-d7tql" pod "d0c2e945-b9c1-40a4-9f2f-e411b9335c46" (UID: "d0c2e945-b9c1-40a4-9f2f-e411b9335c46"). InnerVolumeSpecName "kube-api-access-d7tql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:17.438014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.437968    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7tql\" (UniqueName: \"kubernetes.io/projected/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-kube-api-access-d7tql\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.438195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.438009    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0c2e945-b9c1-40a4-9f2f-e411b9335c46-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.818932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.818891    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j" event={"ID":"d0c2e945-b9c1-40a4-9f2f-e411b9335c46","Type":"ContainerDied","Data":"ff201d9180c5d7acb5a75b0a0203b2d8b21a3fa4bdfa440a2cfff143c0d057e8"}
Oct 30 13:30:17.819498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.818971    4429 scope.go:122] "RemoveContainer" containerID="99589b11f6cdc0575b4b45d281115545e002bd01714d7323c2a978b12f24bcd9"
Oct 30 13:30:17.819498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.819080    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"
Oct 30 13:30:17.822164 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.822115    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" event={"ID":"b0f8d184-a27d-4df3-a63b-42008999091b","Type":"ContainerStarted","Data":"0b013ae5e8d385351f0d65fe01c2bdbd73fb0d139769eca839ac24a3b6950214"}
Oct 30 13:30:17.824453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.824429    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"
Oct 30 13:30:17.824572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.824509    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn" event={"ID":"d5a81ea6-9007-4f05-9780-deaab8f77dde","Type":"ContainerDied","Data":"436fb5bf95217ab3decaa9db1927821353ee304f17a8f0b347a6caa3192852d6"}
Oct 30 13:30:17.831639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.831609    4429 generic.go:356] "Generic (PLEG): container finished" podID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerID="20ffb7d4b43e46f749051ae8dc6b27f6878cab1dc55e497fbc5cf09b706eea23" exitCode=60
Oct 30 13:30:17.831792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.831722    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" event={"ID":"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9","Type":"ContainerDied","Data":"20ffb7d4b43e46f749051ae8dc6b27f6878cab1dc55e497fbc5cf09b706eea23"}
Oct 30 13:30:17.832184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.832165    4429 scope.go:122] "RemoveContainer" containerID="20ffb7d4b43e46f749051ae8dc6b27f6878cab1dc55e497fbc5cf09b706eea23"
Oct 30 13:30:17.832740 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:17.832716    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88_odh-ai-gateway-infra(dd058e91-9991-4df1-91fa-1e2a6e5fbcb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9"
Oct 30 13:30:17.835064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.835048    4429 scope.go:122] "RemoveContainer" containerID="7eb328557203220c87e962c98f7eab93bbfea2f3458edc0cb806b0705636dbc7"
Oct 30 13:30:17.846700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.846683    4429 scope.go:122] "RemoveContainer" containerID="44bf4723cc5dd30f356e9783284ecffe6341dbcb123ad0ab830aa1356dcaf45c"
Oct 30 13:30:17.882979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.882946    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"]
Oct 30 13:30:17.905295 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.905260    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"]
Oct 30 13:30:17.908682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:17.908660    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0d9115-29889450-zrh6j"]
Oct 30 13:30:18.083487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.083416    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" path="/var/lib/kubelet/pods/d0c2e945-b9c1-40a4-9f2f-e411b9335c46/volumes"
Oct 30 13:30:18.086615 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.086590    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"]
Oct 30 13:30:18.092468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.092444    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-bb86fd-29889450-579hn"]
Oct 30 13:30:18.342060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.341979    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-jhgnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 13:30:18.342060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.342032    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 13:30:18.846857 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.846829    4429 generic.go:356] "Generic (PLEG): container finished" podID="b0f8d184-a27d-4df3-a63b-42008999091b" containerID="0b013ae5e8d385351f0d65fe01c2bdbd73fb0d139769eca839ac24a3b6950214" exitCode=60
Oct 30 13:30:18.847255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.846901    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" event={"ID":"b0f8d184-a27d-4df3-a63b-42008999091b","Type":"ContainerDied","Data":"0b013ae5e8d385351f0d65fe01c2bdbd73fb0d139769eca839ac24a3b6950214"}
Oct 30 13:30:18.847255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.846975    4429 scope.go:122] "RemoveContainer" containerID="6687c2ef1cad026b556d8b611abe42add8a14a1f952bf87e5813af5af0696af6"
Oct 30 13:30:18.871313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:18.847487    4429 scope.go:122] "RemoveContainer" containerID="0b013ae5e8d385351f0d65fe01c2bdbd73fb0d139769eca839ac24a3b6950214"
Oct 30 13:30:18.871313 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:18.847895    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx_odh-ai-gateway-infra(b0f8d184-a27d-4df3-a63b-42008999091b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" podUID="b0f8d184-a27d-4df3-a63b-42008999091b"
Oct 30 13:30:19.036423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.036314    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"]
Oct 30 13:30:19.193875 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.193850    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:19.291089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.291064    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-maas-api-service-ca\") pod \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\" (UID: \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\") "
Oct 30 13:30:19.291403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.291389    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-kube-api-access-pngsc\" (UniqueName: \"kubernetes.io/projected/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-kube-api-access-pngsc\") pod \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\" (UID: \"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9\") "
Oct 30 13:30:19.291839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.291815    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-maas-api-service-ca" pod "dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" (UID: "dd058e91-9991-4df1-91fa-1e2a6e5fbcb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:19.292123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.292102    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.294632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.294603    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-kube-api-access-pngsc" pod "dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" (UID: "dd058e91-9991-4df1-91fa-1e2a6e5fbcb9"). InnerVolumeSpecName "kube-api-access-pngsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:19.341384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.341353    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-jhgnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 13:30:19.341551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.341404    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 13:30:19.393752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.393676    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pngsc\" (UniqueName: \"kubernetes.io/projected/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9-kube-api-access-pngsc\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.724739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.724709    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:30:19.895643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.895578    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88" event={"ID":"dd058e91-9991-4df1-91fa-1e2a6e5fbcb9","Type":"ContainerDied","Data":"051cb1a618bb73100f66e149e148a613c3152396eb67b12b9af1142121601845"}
Oct 30 13:30:19.895643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.895651    4429 scope.go:122] "RemoveContainer" containerID="20ffb7d4b43e46f749051ae8dc6b27f6878cab1dc55e497fbc5cf09b706eea23"
Oct 30 13:30:19.896183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.895781    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"
Oct 30 13:30:19.991788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.991109    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"]
Oct 30 13:30:19.997861 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:19.997821    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-bd2aaf-29889450-hzx88"]
Oct 30 13:30:20.079871 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.079841    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" path="/var/lib/kubelet/pods/d5a81ea6-9007-4f05-9780-deaab8f77dde/volumes"
Oct 30 13:30:20.080370 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.080350    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" path="/var/lib/kubelet/pods/dd058e91-9991-4df1-91fa-1e2a6e5fbcb9/volumes"
Oct 30 13:30:20.089278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.089248    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:20.120672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.120646    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f8d184-a27d-4df3-a63b-42008999091b-kube-api-access-4xp7f\" (UniqueName: \"kubernetes.io/projected/b0f8d184-a27d-4df3-a63b-42008999091b-kube-api-access-4xp7f\") pod \"b0f8d184-a27d-4df3-a63b-42008999091b\" (UID: \"b0f8d184-a27d-4df3-a63b-42008999091b\") "
Oct 30 13:30:20.120821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.120712    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f8d184-a27d-4df3-a63b-42008999091b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f8d184-a27d-4df3-a63b-42008999091b-maas-api-service-ca\") pod \"b0f8d184-a27d-4df3-a63b-42008999091b\" (UID: \"b0f8d184-a27d-4df3-a63b-42008999091b\") "
Oct 30 13:30:20.121117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.121092    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f8d184-a27d-4df3-a63b-42008999091b-maas-api-service-ca" pod "b0f8d184-a27d-4df3-a63b-42008999091b" (UID: "b0f8d184-a27d-4df3-a63b-42008999091b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:20.121625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.121598    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f8d184-a27d-4df3-a63b-42008999091b-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.123498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.123470    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f8d184-a27d-4df3-a63b-42008999091b-kube-api-access-4xp7f" pod "b0f8d184-a27d-4df3-a63b-42008999091b" (UID: "b0f8d184-a27d-4df3-a63b-42008999091b"). InnerVolumeSpecName "kube-api-access-4xp7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:20.223192 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.223153    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xp7f\" (UniqueName: \"kubernetes.io/projected/b0f8d184-a27d-4df3-a63b-42008999091b-kube-api-access-4xp7f\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.342147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.342071    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-jhgnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 13:30:20.342147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.342127    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 13:30:20.904819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.904784    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"
Oct 30 13:30:20.904819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.904815    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx" event={"ID":"b0f8d184-a27d-4df3-a63b-42008999091b","Type":"ContainerDied","Data":"3e1baeb9df55148cc09060dd5f80856f4e3f28f524610eae310a348d9ee6fdd7"}
Oct 30 13:30:20.906050 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.904880    4429 scope.go:122] "RemoveContainer" containerID="0b013ae5e8d385351f0d65fe01c2bdbd73fb0d139769eca839ac24a3b6950214"
Oct 30 13:30:20.976741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.976710    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"]
Oct 30 13:30:20.986895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:20.986871    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-88e9f3-29889450-cftbx"]
Oct 30 13:30:21.341936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:21.341905    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-jhgnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 13:30:21.342136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:21.341957    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 13:30:22.080681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:22.080645    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" path="/var/lib/kubelet/pods/b0f8d184-a27d-4df3-a63b-42008999091b/volumes"
Oct 30 13:30:22.491752 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:22.491717    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:22.492716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:22.492692    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:22.494119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:22.494100    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:30:28.065556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.065522    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"]
Oct 30 13:30:28.066342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066326    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:30:28.066384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066343    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:30:28.066384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066353    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:30:28.066384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066358    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:30:28.066384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066373    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:30:28.066384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066378    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066390    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066395    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066400    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066405    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066433    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066437    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066443    4429 cpu_manager.go:414] "containerMap: removing container" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066447    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066465    4429 cpu_manager.go:414] "containerMap: removing container" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066469    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066482    4429 cpu_manager.go:414] "containerMap: removing container" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:30:28.066556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066486    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066584    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066593    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066598    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066603    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066610    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066615    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066619    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066628    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:30:28.066915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.066633    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:30:28.070509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.070492    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.107395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.107363    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"]
Oct 30 13:30:28.147814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.147779    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.147814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.147811    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.148010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.147885    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ce256-b633-48e7-9679-772bc5b6fee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.148010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.147910    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.148010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.148002    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqtpc\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-kube-api-access-hqtpc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.148111 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.148051    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.148111 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.148078    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.148258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.148244    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.148357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.148345    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250049    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250147    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250175    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250231    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ce256-b633-48e7-9679-772bc5b6fee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250263    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250353    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqtpc\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-kube-api-access-hqtpc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250428    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250491    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250648    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250703    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.250940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.250914    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.251159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.251140    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.251312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.251269    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ce256-b633-48e7-9679-772bc5b6fee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.251363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.251340    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.252832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.252814    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.253029 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.253011    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.277813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.277793    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqtpc\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-kube-api-access-hqtpc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.277891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.277817    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-77b6j\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.398225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.398139    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:28.728712 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:30:28.728686    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16ce256_b633_48e7_9679_772bc5b6fee7.slice/crio-f0783a6c2af96f0fd218aeded312080b19dec16f146df211058078ec3c3f6f27 WatchSource:0}: Error finding container f0783a6c2af96f0fd218aeded312080b19dec16f146df211058078ec3c3f6f27: Status 404 returned error can't find the container with id f0783a6c2af96f0fd218aeded312080b19dec16f146df211058078ec3c3f6f27
Oct 30 13:30:28.729751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.729724    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"]
Oct 30 13:30:28.731797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.731765    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:28.731892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.731830    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:28.731892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.731861    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:28.981885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.981784    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" event={"ID":"c16ce256-b633-48e7-9679-772bc5b6fee7","Type":"ContainerStarted","Data":"0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea"}
Oct 30 13:30:28.981885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:28.981848    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" event={"ID":"c16ce256-b633-48e7-9679-772bc5b6fee7","Type":"ContainerStarted","Data":"f0783a6c2af96f0fd218aeded312080b19dec16f146df211058078ec3c3f6f27"}
Oct 30 13:30:29.040070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:29.040015    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podStartSLOduration=1.040000697 podStartE2EDuration="1.040000697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:29.037988262 +0000 UTC m=+3487.093482096" watchObservedRunningTime="2026-10-30 13:30:29.040000697 +0000 UTC m=+3487.095494520"
Oct 30 13:30:29.398675 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:29.398512    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:29.404310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:29.403416    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 13:30:29.404310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:29.403471    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 13:30:30.415308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:30.401890    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 13:30:30.415308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:30.401951    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 13:30:31.399082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:31.399047    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 13:30:31.399267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:31.399101    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 13:30:32.398840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:32.398807    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 13:30:32.399254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:32.398859    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 13:30:33.398593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:33.398561    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 13:30:33.398775 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:33.398612    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 13:30:34.398468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:34.398437    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 13:30:34.398899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:34.398487    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 13:30:35.398793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:35.398756    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 13:30:35.399230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:35.398821    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 13:30:36.398800 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:36.398761    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 13:30:36.399204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:36.398850    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 13:30:38.445315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:38.444440    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-77b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:30:38.445315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:38.444537    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:38.832354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:38.832254    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:38.832665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:38.832641    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:38.833641 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:38.833589    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:30:42.374229 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:42.374190    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:30:52.327966 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:52.327932    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"]
Oct 30 13:30:57.328700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.328663    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx" podUID="44164244-aa4e-45a8-86ca-ffa561bd8724" containerName="payload-processing" containerID="cri-o://e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730" gracePeriod=25
Oct 30 13:30:57.647067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.647042    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:30:57.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.695680    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-kube-api-access-mwb7h\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-kube-api-access-mwb7h\") pod \"44164244-aa4e-45a8-86ca-ffa561bd8724\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") "
Oct 30 13:30:57.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.695754    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-plugins-config-volume\") pod \"44164244-aa4e-45a8-86ca-ffa561bd8724\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") "
Oct 30 13:30:57.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.695977    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-routing-overlay-volume\") pod \"44164244-aa4e-45a8-86ca-ffa561bd8724\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") "
Oct 30 13:30:57.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.696086    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-provider-credentials\") pod \"44164244-aa4e-45a8-86ca-ffa561bd8724\" (UID: \"44164244-aa4e-45a8-86ca-ffa561bd8724\") "
Oct 30 13:30:57.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.696536    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-routing-overlay-volume" pod "44164244-aa4e-45a8-86ca-ffa561bd8724" (UID: "44164244-aa4e-45a8-86ca-ffa561bd8724"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:57.696829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.696807    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:57.697202 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.696995    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-plugins-config-volume" pod "44164244-aa4e-45a8-86ca-ffa561bd8724" (UID: "44164244-aa4e-45a8-86ca-ffa561bd8724"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:57.699397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.699360    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-kube-api-access-mwb7h" pod "44164244-aa4e-45a8-86ca-ffa561bd8724" (UID: "44164244-aa4e-45a8-86ca-ffa561bd8724"). InnerVolumeSpecName "kube-api-access-mwb7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:57.707395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.707371    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-provider-credentials" pod "44164244-aa4e-45a8-86ca-ffa561bd8724" (UID: "44164244-aa4e-45a8-86ca-ffa561bd8724"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:57.797886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.797852    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:57.797886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.797881    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwb7h\" (UniqueName: \"kubernetes.io/projected/44164244-aa4e-45a8-86ca-ffa561bd8724-kube-api-access-mwb7h\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:57.797886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:57.797890    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44164244-aa4e-45a8-86ca-ffa561bd8724-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:30:58.215343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.215277    4429 generic.go:356] "Generic (PLEG): container finished" podID="44164244-aa4e-45a8-86ca-ffa561bd8724" containerID="e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730" exitCode=0
Oct 30 13:30:58.215343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.215324    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx" event={"ID":"44164244-aa4e-45a8-86ca-ffa561bd8724","Type":"ContainerDied","Data":"e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730"}
Oct 30 13:30:58.215553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.215353    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"
Oct 30 13:30:58.215553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.215383    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx" event={"ID":"44164244-aa4e-45a8-86ca-ffa561bd8724","Type":"ContainerDied","Data":"71fb4c1f09ebb0a7a4ef0b351ac1d480ba025b1a13b3825cd9830db443555f4e"}
Oct 30 13:30:58.215553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.215406    4429 scope.go:122] "RemoveContainer" containerID="e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730"
Oct 30 13:30:58.225795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.225772    4429 scope.go:122] "RemoveContainer" containerID="e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730"
Oct 30 13:30:58.226112 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:30:58.226086    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730\": container with ID starting with e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730 not found: ID does not exist" containerID="e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730"
Oct 30 13:30:58.226232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.226120    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730"} err="failed to get container status \"e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730\": rpc error: code = NotFound desc = could not find container \"e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730\": container with ID starting with e4d6e2f322e1c64db17926dbc8f72333ac8a8dd7c8ce122a5701ba3eba315730 not found: ID does not exist"
Oct 30 13:30:58.276503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.276469    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"]
Oct 30 13:30:58.290071 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:30:58.290046    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-5766z8vvx"]
Oct 30 13:31:00.077479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:00.077446    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44164244-aa4e-45a8-86ca-ffa561bd8724" path="/var/lib/kubelet/pods/44164244-aa4e-45a8-86ca-ffa561bd8724/volumes"
Oct 30 13:31:12.372357 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:31:12.372315    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:31:12.867521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.867492    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"]
Oct 30 13:31:12.868242 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868228    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:31:12.868331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868244    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:31:12.868331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868266    4429 cpu_manager.go:414] "containerMap: removing container" podUID="44164244-aa4e-45a8-86ca-ffa561bd8724" containerName="payload-processing"
Oct 30 13:31:12.868331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868271    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44164244-aa4e-45a8-86ca-ffa561bd8724" containerName="payload-processing"
Oct 30 13:31:12.868331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868315    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:31:12.868331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868321    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:31:12.868511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868344    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:31:12.868511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868350    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a81ea6-9007-4f05-9780-deaab8f77dde" containerName="cleanup"
Oct 30 13:31:12.868511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868465    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd058e91-9991-4df1-91fa-1e2a6e5fbcb9" containerName="cleanup"
Oct 30 13:31:12.868511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868478    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c2e945-b9c1-40a4-9f2f-e411b9335c46" containerName="cleanup"
Oct 30 13:31:12.868511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868486    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="44164244-aa4e-45a8-86ca-ffa561bd8724" containerName="payload-processing"
Oct 30 13:31:12.868511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.868497    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f8d184-a27d-4df3-a63b-42008999091b" containerName="cleanup"
Oct 30 13:31:12.873121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.873101    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:12.891621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.891595    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"]
Oct 30 13:31:12.901459 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.901434    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"openshift-service-ca.crt\""
Oct 30 13:31:12.901619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.901604    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"kube-root-ca.crt\""
Oct 30 13:31:12.901702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.901680    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"routing-overlay\""
Oct 30 13:31:12.901740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.901729    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1957dc\""
Oct 30 13:31:12.901782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.901769    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"agc-cred-adafb933cb9f\""
Oct 30 13:31:12.903561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.903544    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1957dc\"/\"payload-processing-external-model-e2e--8217e17a-dockercfg-v72th\""
Oct 30 13:31:12.917060 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.917036    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:12.917180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.917096    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qf9\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-kube-api-access-d5qf9\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:12.917312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.917274    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:12.917523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:12.917502    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.026323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.026263    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.026778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.026753    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.027022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.027004    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.027175 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.027142    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qf9\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-kube-api-access-d5qf9\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.028598 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.028575    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.043401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.042398    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.053692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.050765    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.065344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.064179    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qf9\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-kube-api-access-d5qf9\") pod \"payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") " pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.189895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.189863    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:13.510241 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:31:13.510203    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f33746_beac_461c_ab0f_55d8735b8b67.slice/crio-03ea3cf4c4a5d059457f0c798933794a5c06c1da1e1b34f39afe7e33152d9cf7 WatchSource:0}: Error finding container 03ea3cf4c4a5d059457f0c798933794a5c06c1da1e1b34f39afe7e33152d9cf7: Status 404 returned error can't find the container with id 03ea3cf4c4a5d059457f0c798933794a5c06c1da1e1b34f39afe7e33152d9cf7
Oct 30 13:31:13.511000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.510975    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"]
Oct 30 13:31:13.851853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.851777    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"]
Oct 30 13:31:13.856582 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.856564    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:13.873585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.873062    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-d3093dc79fb2\""
Oct 30 13:31:13.873585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.873503    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-c4rvp\""
Oct 30 13:31:13.873982 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.873959    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"]
Oct 30 13:31:13.883664 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.883638    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\""
Oct 30 13:31:13.883809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.883792    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\""
Oct 30 13:31:13.885029 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.885008    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\""
Oct 30 13:31:13.885146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.885091    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\""
Oct 30 13:31:13.945989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.945956    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:13.946165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.946019    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdqrk\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-kube-api-access-wdqrk\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:13.946165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.946143    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:13.946392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:13.946373    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.061881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.060793    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.061881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.060874    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdqrk\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-kube-api-access-wdqrk\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.061881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.060928    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.061881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.061264    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.064093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.062958    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.064093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.063865    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.081273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.078557    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.087420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.087393    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdqrk\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-kube-api-access-wdqrk\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.172642 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.172566    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:14.339777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.339739    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7" event={"ID":"65f33746-beac-461c-ab0f-55d8735b8b67","Type":"ContainerStarted","Data":"7883dbfc03b09fc02b81b4ef4ce656f2fd42dbcd65bd7c9fff94508903b13cb5"}
Oct 30 13:31:14.339973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.339790    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7" event={"ID":"65f33746-beac-461c-ab0f-55d8735b8b67","Type":"ContainerStarted","Data":"03ea3cf4c4a5d059457f0c798933794a5c06c1da1e1b34f39afe7e33152d9cf7"}
Oct 30 13:31:14.339973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.339817    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:14.387809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.387757    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7" podStartSLOduration=2.387742529 podStartE2EDuration="2.387742529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:14.38395637 +0000 UTC m=+3532.439450194" watchObservedRunningTime="2026-10-30 13:31:14.387742529 +0000 UTC m=+3532.443236385"
Oct 30 13:31:14.476518 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:31:14.476489    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033d112b_7ac6_46d9_91c8_3ffa5be709be.slice/crio-5c9454161412e45145f502332c229f831d2d51e842239823ce79f278f4b36da7 WatchSource:0}: Error finding container 5c9454161412e45145f502332c229f831d2d51e842239823ce79f278f4b36da7: Status 404 returned error can't find the container with id 5c9454161412e45145f502332c229f831d2d51e842239823ce79f278f4b36da7
Oct 30 13:31:14.479678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:14.479645    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"]
Oct 30 13:31:15.349863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:15.349827    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps" event={"ID":"033d112b-7ac6-46d9-91c8-3ffa5be709be","Type":"ContainerStarted","Data":"f36762dc79362e81409640d5667947a41a913e89a962dba8683f9ed9b7e4cfdb"}
Oct 30 13:31:15.350362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:15.349893    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps" event={"ID":"033d112b-7ac6-46d9-91c8-3ffa5be709be","Type":"ContainerStarted","Data":"5c9454161412e45145f502332c229f831d2d51e842239823ce79f278f4b36da7"}
Oct 30 13:31:15.350362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:15.350125    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:15.397325 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:15.397257    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps" podStartSLOduration=2.39724387 podStartE2EDuration="2.39724387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:15.395232314 +0000 UTC m=+3533.450726137" watchObservedRunningTime="2026-10-30 13:31:15.39724387 +0000 UTC m=+3533.452737695"
Oct 30 13:31:25.353304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:25.353259    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:26.385547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:26.385513    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:37.350342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:37.350233    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"]
Oct 30 13:31:39.365138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:39.365105    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"]
Oct 30 13:31:42.351370 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.351324    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps" podUID="033d112b-7ac6-46d9-91c8-3ffa5be709be" containerName="payload-processing" containerID="cri-o://f36762dc79362e81409640d5667947a41a913e89a962dba8683f9ed9b7e4cfdb" gracePeriod=25
Oct 30 13:31:42.375497 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:31:42.375461    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:31:42.590566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.590515    4429 generic.go:356] "Generic (PLEG): container finished" podID="033d112b-7ac6-46d9-91c8-3ffa5be709be" containerID="f36762dc79362e81409640d5667947a41a913e89a962dba8683f9ed9b7e4cfdb" exitCode=0
Oct 30 13:31:42.590763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.590589    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps" event={"ID":"033d112b-7ac6-46d9-91c8-3ffa5be709be","Type":"ContainerDied","Data":"f36762dc79362e81409640d5667947a41a913e89a962dba8683f9ed9b7e4cfdb"}
Oct 30 13:31:42.626129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.626103    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:42.761487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.761460    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-kube-api-access-wdqrk\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-kube-api-access-wdqrk\") pod \"033d112b-7ac6-46d9-91c8-3ffa5be709be\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") "
Oct 30 13:31:42.761667 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.761522    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-plugins-config-volume\") pod \"033d112b-7ac6-46d9-91c8-3ffa5be709be\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") "
Oct 30 13:31:42.761743 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.761715    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-provider-credentials\") pod \"033d112b-7ac6-46d9-91c8-3ffa5be709be\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") "
Oct 30 13:31:42.761832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.761817    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-routing-overlay-volume\") pod \"033d112b-7ac6-46d9-91c8-3ffa5be709be\" (UID: \"033d112b-7ac6-46d9-91c8-3ffa5be709be\") "
Oct 30 13:31:42.762123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.762089    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-plugins-config-volume" pod "033d112b-7ac6-46d9-91c8-3ffa5be709be" (UID: "033d112b-7ac6-46d9-91c8-3ffa5be709be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:42.762262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.762159    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-routing-overlay-volume" pod "033d112b-7ac6-46d9-91c8-3ffa5be709be" (UID: "033d112b-7ac6-46d9-91c8-3ffa5be709be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:42.762567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.762550    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:31:42.762614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.762573    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/033d112b-7ac6-46d9-91c8-3ffa5be709be-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:31:42.764018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.763997    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-kube-api-access-wdqrk" pod "033d112b-7ac6-46d9-91c8-3ffa5be709be" (UID: "033d112b-7ac6-46d9-91c8-3ffa5be709be"). InnerVolumeSpecName "kube-api-access-wdqrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:42.764131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.764045    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-provider-credentials" pod "033d112b-7ac6-46d9-91c8-3ffa5be709be" (UID: "033d112b-7ac6-46d9-91c8-3ffa5be709be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:42.864337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.864241    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:31:42.864337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:42.864269    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdqrk\" (UniqueName: \"kubernetes.io/projected/033d112b-7ac6-46d9-91c8-3ffa5be709be-kube-api-access-wdqrk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:31:43.600107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:43.600079    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"
Oct 30 13:31:43.600633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:43.600116    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps" event={"ID":"033d112b-7ac6-46d9-91c8-3ffa5be709be","Type":"ContainerDied","Data":"5c9454161412e45145f502332c229f831d2d51e842239823ce79f278f4b36da7"}
Oct 30 13:31:43.600633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:43.600178    4429 scope.go:122] "RemoveContainer" containerID="f36762dc79362e81409640d5667947a41a913e89a962dba8683f9ed9b7e4cfdb"
Oct 30 13:31:43.658701 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:43.658673    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"]
Oct 30 13:31:43.671990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:43.671966    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c896r4ps"]
Oct 30 13:31:44.076246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.076211    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033d112b-7ac6-46d9-91c8-3ffa5be709be" path="/var/lib/kubelet/pods/033d112b-7ac6-46d9-91c8-3ffa5be709be/volumes"
Oct 30 13:31:44.366210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.366126    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7" podUID="65f33746-beac-461c-ab0f-55d8735b8b67" containerName="payload-processing" containerID="cri-o://7883dbfc03b09fc02b81b4ef4ce656f2fd42dbcd65bd7c9fff94508903b13cb5" gracePeriod=25
Oct 30 13:31:44.608486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.608454    4429 generic.go:356] "Generic (PLEG): container finished" podID="65f33746-beac-461c-ab0f-55d8735b8b67" containerID="7883dbfc03b09fc02b81b4ef4ce656f2fd42dbcd65bd7c9fff94508903b13cb5" exitCode=0
Oct 30 13:31:44.608970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.608528    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7" event={"ID":"65f33746-beac-461c-ab0f-55d8735b8b67","Type":"ContainerDied","Data":"7883dbfc03b09fc02b81b4ef4ce656f2fd42dbcd65bd7c9fff94508903b13cb5"}
Oct 30 13:31:44.655244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.655221    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:44.789205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.789176    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-provider-credentials\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-provider-credentials\") pod \"65f33746-beac-461c-ab0f-55d8735b8b67\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") "
Oct 30 13:31:44.789436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.789308    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-kube-api-access-d5qf9\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-kube-api-access-d5qf9\") pod \"65f33746-beac-461c-ab0f-55d8735b8b67\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") "
Oct 30 13:31:44.789436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.789352    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-routing-overlay-volume\") pod \"65f33746-beac-461c-ab0f-55d8735b8b67\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") "
Oct 30 13:31:44.789574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.789496    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-plugins-config-volume\") pod \"65f33746-beac-461c-ab0f-55d8735b8b67\" (UID: \"65f33746-beac-461c-ab0f-55d8735b8b67\") "
Oct 30 13:31:44.789830 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.789798    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-routing-overlay-volume" pod "65f33746-beac-461c-ab0f-55d8735b8b67" (UID: "65f33746-beac-461c-ab0f-55d8735b8b67"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:44.789970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.789913    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-plugins-config-volume" pod "65f33746-beac-461c-ab0f-55d8735b8b67" (UID: "65f33746-beac-461c-ab0f-55d8735b8b67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:44.790324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.790298    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:31:44.790324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.790322    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f33746-beac-461c-ab0f-55d8735b8b67-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:31:44.791709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.791691    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-provider-credentials" pod "65f33746-beac-461c-ab0f-55d8735b8b67" (UID: "65f33746-beac-461c-ab0f-55d8735b8b67"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:44.792131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.792110    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-kube-api-access-d5qf9" pod "65f33746-beac-461c-ab0f-55d8735b8b67" (UID: "65f33746-beac-461c-ab0f-55d8735b8b67"). InnerVolumeSpecName "kube-api-access-d5qf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:44.891468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.891387    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qf9\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-kube-api-access-d5qf9\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:31:44.891468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:44.891424    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65f33746-beac-461c-ab0f-55d8735b8b67-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:31:45.617681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:45.617649    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7" event={"ID":"65f33746-beac-461c-ab0f-55d8735b8b67","Type":"ContainerDied","Data":"03ea3cf4c4a5d059457f0c798933794a5c06c1da1e1b34f39afe7e33152d9cf7"}
Oct 30 13:31:45.618166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:45.617706    4429 scope.go:122] "RemoveContainer" containerID="7883dbfc03b09fc02b81b4ef4ce656f2fd42dbcd65bd7c9fff94508903b13cb5"
Oct 30 13:31:45.618166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:45.617654    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"
Oct 30 13:31:45.680117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:45.680088    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"]
Oct 30 13:31:45.693347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:45.693326    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1957dc/payload-processing-external-model-e2e-extmod-a-1957dc-7f7dqzkl7"]
Oct 30 13:31:46.075966 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:31:46.075933    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f33746-beac-461c-ab0f-55d8735b8b67" path="/var/lib/kubelet/pods/65f33746-beac-461c-ab0f-55d8735b8b67/volumes"
Oct 30 13:32:12.373673 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:32:12.373633    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:32:27.376994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.376957    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"]
Oct 30 13:32:27.377780 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.377763    4429 cpu_manager.go:414] "containerMap: removing container" podUID="65f33746-beac-461c-ab0f-55d8735b8b67" containerName="payload-processing"
Oct 30 13:32:27.377831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.377781    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f33746-beac-461c-ab0f-55d8735b8b67" containerName="payload-processing"
Oct 30 13:32:27.377831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.377806    4429 cpu_manager.go:414] "containerMap: removing container" podUID="033d112b-7ac6-46d9-91c8-3ffa5be709be" containerName="payload-processing"
Oct 30 13:32:27.377831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.377811    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033d112b-7ac6-46d9-91c8-3ffa5be709be" containerName="payload-processing"
Oct 30 13:32:27.377959 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.377948    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f33746-beac-461c-ab0f-55d8735b8b67" containerName="payload-processing"
Oct 30 13:32:27.377996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.377963    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="033d112b-7ac6-46d9-91c8-3ffa5be709be" containerName="payload-processing"
Oct 30 13:32:27.381773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.381754    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:27.402613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.402590    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"]
Oct 30 13:32:27.412530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.412496    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d41320\""
Oct 30 13:32:27.412666 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.412538    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d41320-dockercfg-gdfw7\""
Oct 30 13:32:27.500305 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.486274    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f35f65b-2202-4bd9-b8ea-705c55d37960-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d\" (UID: \"4f35f65b-2202-4bd9-b8ea-705c55d37960\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:27.500305 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.486482    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt8jb\" (UniqueName: \"kubernetes.io/projected/4f35f65b-2202-4bd9-b8ea-705c55d37960-kube-api-access-vt8jb\") pod \"payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d\" (UID: \"4f35f65b-2202-4bd9-b8ea-705c55d37960\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:27.560692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.560652    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"]
Oct 30 13:32:27.592637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.592207    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt8jb\" (UniqueName: \"kubernetes.io/projected/4f35f65b-2202-4bd9-b8ea-705c55d37960-kube-api-access-vt8jb\") pod \"payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d\" (UID: \"4f35f65b-2202-4bd9-b8ea-705c55d37960\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:27.593144 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.593107    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f35f65b-2202-4bd9-b8ea-705c55d37960-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d\" (UID: \"4f35f65b-2202-4bd9-b8ea-705c55d37960\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:27.594323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.594107    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f35f65b-2202-4bd9-b8ea-705c55d37960-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d\" (UID: \"4f35f65b-2202-4bd9-b8ea-705c55d37960\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:27.639783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.638946    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt8jb\" (UniqueName: \"kubernetes.io/projected/4f35f65b-2202-4bd9-b8ea-705c55d37960-kube-api-access-vt8jb\") pod \"payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d\" (UID: \"4f35f65b-2202-4bd9-b8ea-705c55d37960\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:27.662692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.662664    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"]
Oct 30 13:32:27.662888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.662779    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:27.695882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.695854    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbw7\" (UniqueName: \"kubernetes.io/projected/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-kube-api-access-lmbw7\") pod \"payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg\" (UID: \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:27.696122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.696100    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg\" (UID: \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:27.696358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.696343    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:27.797013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.796983    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg\" (UID: \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:27.797205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.797082    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbw7\" (UniqueName: \"kubernetes.io/projected/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-kube-api-access-lmbw7\") pod \"payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg\" (UID: \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:27.798234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.798204    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg\" (UID: \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:27.866534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.866504    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbw7\" (UniqueName: \"kubernetes.io/projected/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-kube-api-access-lmbw7\") pod \"payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg\" (UID: \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:27.980038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:27.980009    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:28.011860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:28.011725    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"]
Oct 30 13:32:28.016718 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:32:28.016680    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f35f65b_2202_4bd9_b8ea_705c55d37960.slice/crio-891bc1ec8c5d9b30f6345c4e7c57bfb566ad614793dfeaa3e58180c7f302274b WatchSource:0}: Error finding container 891bc1ec8c5d9b30f6345c4e7c57bfb566ad614793dfeaa3e58180c7f302274b: Status 404 returned error can't find the container with id 891bc1ec8c5d9b30f6345c4e7c57bfb566ad614793dfeaa3e58180c7f302274b
Oct 30 13:32:28.310468 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:32:28.310435    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad53602_bf0d_4ca4_9261_e515e8b05e4e.slice/crio-1c289d3bc5252a9010d9ab01dacb278ba1c3e020c0f2f0a1724340ae977b524d WatchSource:0}: Error finding container 1c289d3bc5252a9010d9ab01dacb278ba1c3e020c0f2f0a1724340ae977b524d: Status 404 returned error can't find the container with id 1c289d3bc5252a9010d9ab01dacb278ba1c3e020c0f2f0a1724340ae977b524d
Oct 30 13:32:28.312583 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:28.312556    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"]
Oct 30 13:32:29.021203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:29.021169    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d" event={"ID":"4f35f65b-2202-4bd9-b8ea-705c55d37960","Type":"ContainerStarted","Data":"c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1"}
Oct 30 13:32:29.021730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:29.021233    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:29.021730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:29.021258    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d" event={"ID":"4f35f65b-2202-4bd9-b8ea-705c55d37960","Type":"ContainerStarted","Data":"891bc1ec8c5d9b30f6345c4e7c57bfb566ad614793dfeaa3e58180c7f302274b"}
Oct 30 13:32:29.022992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:29.022967    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg" event={"ID":"9ad53602-bf0d-4ca4-9261-e515e8b05e4e","Type":"ContainerStarted","Data":"29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3"}
Oct 30 13:32:29.023116 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:29.023017    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg" event={"ID":"9ad53602-bf0d-4ca4-9261-e515e8b05e4e","Type":"ContainerStarted","Data":"1c289d3bc5252a9010d9ab01dacb278ba1c3e020c0f2f0a1724340ae977b524d"}
Oct 30 13:32:29.023116 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:29.023047    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:29.078120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:29.078065    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d" podStartSLOduration=2.078053157 podStartE2EDuration="2.078053157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:29.077637486 +0000 UTC m=+3607.133131309" watchObservedRunningTime="2026-10-30 13:32:29.078053157 +0000 UTC m=+3607.133547050"
Oct 30 13:32:29.125586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:29.125536    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg" podStartSLOduration=2.125521623 podStartE2EDuration="2.125521623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:29.121922063 +0000 UTC m=+3607.177415887" watchObservedRunningTime="2026-10-30 13:32:29.125521623 +0000 UTC m=+3607.181015479"
Oct 30 13:32:40.033872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:40.033842    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:32:40.034308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:40.033913    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:32:42.375095 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:32:42.375047    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:32:54.493953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.493916    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"]
Oct 30 13:32:54.499062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.499045    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.532651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.528800    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-75d385\"/\"agc-cred-1bd41ac28c7a\""
Oct 30 13:32:54.533446 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.533402    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"]
Oct 30 13:32:54.583363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.583325    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.583535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.583447    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5f86\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-kube-api-access-n5f86\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.583613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.583597    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.583668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.583657    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.685026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.684995    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.685184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.685152    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.685244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.685230    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5f86\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-kube-api-access-n5f86\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.685401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.685386    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.685933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.685906    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.686187 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.686166    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.687769 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.687749    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.712552 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.712524    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5f86\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-kube-api-access-n5f86\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:54.822541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:54.822473    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:55.128752 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:32:55.128723    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc9a7f5_4bed_4e75_ac09_d99b8c2b9247.slice/crio-7b67718b54b7634d2b9403f684704b2b8b5a6678862fcf547aa211aba23c29a7 WatchSource:0}: Error finding container 7b67718b54b7634d2b9403f684704b2b8b5a6678862fcf547aa211aba23c29a7: Status 404 returned error can't find the container with id 7b67718b54b7634d2b9403f684704b2b8b5a6678862fcf547aa211aba23c29a7
Oct 30 13:32:55.129874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.129851    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"]
Oct 30 13:32:55.252054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.252025    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b" event={"ID":"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247","Type":"ContainerStarted","Data":"c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79"}
Oct 30 13:32:55.252228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.252077    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b" event={"ID":"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247","Type":"ContainerStarted","Data":"7b67718b54b7634d2b9403f684704b2b8b5a6678862fcf547aa211aba23c29a7"}
Oct 30 13:32:55.252228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.252100    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:32:55.299652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.299609    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b" podStartSLOduration=1.2995962890000001 podStartE2EDuration="1.299596289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:55.299500905 +0000 UTC m=+3633.354994747" watchObservedRunningTime="2026-10-30 13:32:55.299596289 +0000 UTC m=+3633.355090113"
Oct 30 13:32:55.325019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.324984    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"]
Oct 30 13:32:55.329780 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.329764    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.349595 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.349570    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"]
Oct 30 13:32:55.349754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.349734    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"agc-cred-32d76723ab73\""
Oct 30 13:32:55.349806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.349770    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"agc-cred-7ff28565225a\""
Oct 30 13:32:55.350040 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.350022    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-e2e--1b2e61a1-dockercfg-6d4mp\""
Oct 30 13:32:55.359443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.359418    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"openshift-service-ca.crt\""
Oct 30 13:32:55.359600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.359579    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4db150\""
Oct 30 13:32:55.359712 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.359696    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"kube-root-ca.crt\""
Oct 30 13:32:55.359774 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.359760    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4db150\"/\"routing-overlay\""
Oct 30 13:32:55.395768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.395697    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.395768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.395733    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtqmx\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-kube-api-access-wtqmx\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.396004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.395983    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.396126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.396112    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.497991 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.497955    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.498572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.498052    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.498572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.498142    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.498572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.498167    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtqmx\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-kube-api-access-wtqmx\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.498893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.498871    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.499024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.499003    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.500712 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.500693    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.523412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.523388    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtqmx\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-kube-api-access-wtqmx\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.644258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.644227    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:55.958900 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:32:55.952437    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23fd937_ce25_4779_9cb8_d55e6486a656.slice/crio-5309cdf4fa43bb80fa9b095c752b31b48f8b266d4d53cf5a6b78fdf2ede85590 WatchSource:0}: Error finding container 5309cdf4fa43bb80fa9b095c752b31b48f8b266d4d53cf5a6b78fdf2ede85590: Status 404 returned error can't find the container with id 5309cdf4fa43bb80fa9b095c752b31b48f8b266d4d53cf5a6b78fdf2ede85590
Oct 30 13:32:55.958900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:55.954644    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"]
Oct 30 13:32:56.261297 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:56.261183    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx" event={"ID":"c23fd937-ce25-4779-9cb8-d55e6486a656","Type":"ContainerStarted","Data":"f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f"}
Oct 30 13:32:56.261297 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:56.261253    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx" event={"ID":"c23fd937-ce25-4779-9cb8-d55e6486a656","Type":"ContainerStarted","Data":"5309cdf4fa43bb80fa9b095c752b31b48f8b266d4d53cf5a6b78fdf2ede85590"}
Oct 30 13:32:56.261544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:56.261312    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:32:56.309609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:32:56.309557    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx" podStartSLOduration=1.309538257 podStartE2EDuration="1.309538257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:56.30596007 +0000 UTC m=+3634.361453911" watchObservedRunningTime="2026-10-30 13:32:56.309538257 +0000 UTC m=+3634.365032080"
Oct 30 13:33:06.264582 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:06.264506    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:33:06.365876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:06.365830    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"]
Oct 30 13:33:07.271718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:07.271689    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:33:11.366910 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.366847    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q" podUID="d89106ea-620c-4241-a90c-c449c4d866fb" containerName="payload-processing" containerID="cri-o://33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55" gracePeriod=25
Oct 30 13:33:11.661039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.661010    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:33:11.747737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.747697    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-provider-credentials\") pod \"d89106ea-620c-4241-a90c-c449c4d866fb\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") "
Oct 30 13:33:11.748006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.747907    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-kube-api-access-pwvtd\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-kube-api-access-pwvtd\") pod \"d89106ea-620c-4241-a90c-c449c4d866fb\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") "
Oct 30 13:33:11.748006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.747992    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-plugins-config-volume\") pod \"d89106ea-620c-4241-a90c-c449c4d866fb\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") "
Oct 30 13:33:11.748139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.748077    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-routing-overlay-volume\") pod \"d89106ea-620c-4241-a90c-c449c4d866fb\" (UID: \"d89106ea-620c-4241-a90c-c449c4d866fb\") "
Oct 30 13:33:11.749010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.748979    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-routing-overlay-volume" pod "d89106ea-620c-4241-a90c-c449c4d866fb" (UID: "d89106ea-620c-4241-a90c-c449c4d866fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:11.750010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.749980    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-plugins-config-volume" pod "d89106ea-620c-4241-a90c-c449c4d866fb" (UID: "d89106ea-620c-4241-a90c-c449c4d866fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:11.757652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.757617    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-provider-credentials" pod "d89106ea-620c-4241-a90c-c449c4d866fb" (UID: "d89106ea-620c-4241-a90c-c449c4d866fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:11.776440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.775514    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-kube-api-access-pwvtd" pod "d89106ea-620c-4241-a90c-c449c4d866fb" (UID: "d89106ea-620c-4241-a90c-c449c4d866fb"). InnerVolumeSpecName "kube-api-access-pwvtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:11.849964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.849933    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:33:11.849964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.849961    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:33:11.849964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.849971    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwvtd\" (UniqueName: \"kubernetes.io/projected/d89106ea-620c-4241-a90c-c449c4d866fb-kube-api-access-pwvtd\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:33:11.850195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.849979    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89106ea-620c-4241-a90c-c449c4d866fb-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:33:11.992129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.992095    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"]
Oct 30 13:33:11.992934 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.992917    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d89106ea-620c-4241-a90c-c449c4d866fb" containerName="payload-processing"
Oct 30 13:33:11.993005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.992935    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89106ea-620c-4241-a90c-c449c4d866fb" containerName="payload-processing"
Oct 30 13:33:11.993061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.993055    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89106ea-620c-4241-a90c-c449c4d866fb" containerName="payload-processing"
Oct 30 13:33:11.997129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:11.997113    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.017545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.017513    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"]
Oct 30 13:33:12.026520 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.026501    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d41320-serving-cert\""
Oct 30 13:33:12.052172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.052130    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3af666-4cad-4951-8284-0daf00f7eb3d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.052409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.052182    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzsbp\" (UniqueName: \"kubernetes.io/projected/ef3af666-4cad-4951-8284-0daf00f7eb3d-kube-api-access-kzsbp\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.052409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.052332    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-maas-api-tls\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.052636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.052464    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-metrics-certs\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.153775 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.153748    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-maas-api-tls\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.153900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.153792    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-metrics-certs\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.154265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.154234    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3af666-4cad-4951-8284-0daf00f7eb3d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.154420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.154294    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsbp\" (UniqueName: \"kubernetes.io/projected/ef3af666-4cad-4951-8284-0daf00f7eb3d-kube-api-access-kzsbp\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.154979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.154957    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3af666-4cad-4951-8284-0daf00f7eb3d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.156215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.156192    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-maas-api-tls\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.156328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.156313    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-metrics-certs\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.179373 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.179351    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzsbp\" (UniqueName: \"kubernetes.io/projected/ef3af666-4cad-4951-8284-0daf00f7eb3d-kube-api-access-kzsbp\") pod \"maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.312406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.312311    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:12.396090 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:33:12.381550    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:33:12.404026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.403102    4429 generic.go:356] "Generic (PLEG): container finished" podID="d89106ea-620c-4241-a90c-c449c4d866fb" containerID="33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55" exitCode=0
Oct 30 13:33:12.404026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.403164    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"
Oct 30 13:33:12.404026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.403391    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q" event={"ID":"d89106ea-620c-4241-a90c-c449c4d866fb","Type":"ContainerDied","Data":"33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55"}
Oct 30 13:33:12.404026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.403441    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q" event={"ID":"d89106ea-620c-4241-a90c-c449c4d866fb","Type":"ContainerDied","Data":"0a60130ba78e1be7a07987c57cf2734bff7c0feefd845c18ecc0b81ab3306a62"}
Oct 30 13:33:12.404026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.403470    4429 scope.go:122] "RemoveContainer" containerID="33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55"
Oct 30 13:33:12.426682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.425210    4429 scope.go:122] "RemoveContainer" containerID="33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55"
Oct 30 13:33:12.426682 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:33:12.425938    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55\": container with ID starting with 33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55 not found: ID does not exist" containerID="33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55"
Oct 30 13:33:12.426682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.425973    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55"} err="failed to get container status \"33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55\": rpc error: code = NotFound desc = could not find container \"33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55\": container with ID starting with 33a197e07a97416e02d8ee062e55c59b40c5589eb08a06a3a929ee3539849c55 not found: ID does not exist"
Oct 30 13:33:12.466975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.466954    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"]
Oct 30 13:33:12.480407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.480386    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5p6k7q"]
Oct 30 13:33:12.619098 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:33:12.619068    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3af666_4cad_4951_8284_0daf00f7eb3d.slice/crio-8717628a9e731343e0e59cf27c3d139dba952d8f1e0107c40a283a7962407e3e WatchSource:0}: Error finding container 8717628a9e731343e0e59cf27c3d139dba952d8f1e0107c40a283a7962407e3e: Status 404 returned error can't find the container with id 8717628a9e731343e0e59cf27c3d139dba952d8f1e0107c40a283a7962407e3e
Oct 30 13:33:12.619744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.619642    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"]
Oct 30 13:33:12.620808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:12.620789    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:33:13.426711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:13.426613    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7" event={"ID":"ef3af666-4cad-4951-8284-0daf00f7eb3d","Type":"ContainerStarted","Data":"4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b"}
Oct 30 13:33:13.427182 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:13.426715    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7" event={"ID":"ef3af666-4cad-4951-8284-0daf00f7eb3d","Type":"ContainerStarted","Data":"8717628a9e731343e0e59cf27c3d139dba952d8f1e0107c40a283a7962407e3e"}
Oct 30 13:33:13.427182 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:13.426845    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:13.474238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:13.474192    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7" podStartSLOduration=2.095859359 podStartE2EDuration="2.4741782s" totalImagesPullingTime="378.318841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:13.472373815 +0000 UTC m=+3651.527867658" watchObservedRunningTime="2026-10-30 13:33:13.4741782 +0000 UTC m=+3651.529672023"
Oct 30 13:33:14.076678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:14.076632    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89106ea-620c-4241-a90c-c449c4d866fb" path="/var/lib/kubelet/pods/d89106ea-620c-4241-a90c-c449c4d866fb/volumes"
Oct 30 13:33:19.440256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:19.440223    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:33:31.327810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.327772    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"]
Oct 30 13:33:31.332631 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.332607    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.351916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.351874    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"]
Oct 30 13:33:31.363481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.363445    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"payload-processing-external-model-e2e--66ed339f-dockercfg-sh52l\""
Oct 30 13:33:31.363768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.363745    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"openshift-service-ca.crt\""
Oct 30 13:33:31.364443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.364415    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"kube-root-ca.crt\""
Oct 30 13:33:31.364583 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.364491    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"agc-cred-ce4f2c5394da\""
Oct 30 13:33:31.364583 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.364527    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"agc-cred-7337afd955bc\""
Oct 30 13:33:31.364706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.364594    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"routing-overlay\""
Oct 30 13:33:31.364706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.364693    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-acf9e2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-acf9e2\""
Oct 30 13:33:31.394997 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.394957    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.395225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.395107    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.395331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.395227    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8fjp\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-kube-api-access-j8fjp\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.395413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.395397    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.496867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.496831    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.497089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.496953    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.497089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.497036    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.497233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.497109    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8fjp\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-kube-api-access-j8fjp\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.498215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.498181    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.498215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.498200    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.499758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.499740    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.523616 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.523576    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8fjp\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-kube-api-access-j8fjp\") pod \"payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") " pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.650567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.650464    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:31.961454 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:33:31.961419    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae856cfe_bcfd_4cc5_b902_54a85b7cb9d6.slice/crio-34053a4bf51e28de7d7423610aa12afa466616783f376a1d752853b4c1494b0e WatchSource:0}: Error finding container 34053a4bf51e28de7d7423610aa12afa466616783f376a1d752853b4c1494b0e: Status 404 returned error can't find the container with id 34053a4bf51e28de7d7423610aa12afa466616783f376a1d752853b4c1494b0e
Oct 30 13:33:31.963956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:31.963926    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"]
Oct 30 13:33:32.593614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:32.593570    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm" event={"ID":"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6","Type":"ContainerStarted","Data":"363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462"}
Oct 30 13:33:32.594136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:32.593649    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:32.594136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:32.593675    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm" event={"ID":"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6","Type":"ContainerStarted","Data":"34053a4bf51e28de7d7423610aa12afa466616783f376a1d752853b4c1494b0e"}
Oct 30 13:33:32.645217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:32.645162    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm" podStartSLOduration=1.645147025 podStartE2EDuration="1.645147025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:32.644087487 +0000 UTC m=+3670.699581315" watchObservedRunningTime="2026-10-30 13:33:32.645147025 +0000 UTC m=+3670.700640849"
Oct 30 13:33:39.311236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:39.311200    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"]
Oct 30 13:33:39.315000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:39.314968    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:42.377119 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:33:42.377079    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:33:44.312386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.312338    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm" podUID="ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" containerName="payload-processing" containerID="cri-o://363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462" gracePeriod=25
Oct 30 13:33:44.598710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.598684    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:44.629411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.629375    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-provider-credentials\") pod \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") "
Oct 30 13:33:44.629617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.629517    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-kube-api-access-j8fjp\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-kube-api-access-j8fjp\") pod \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") "
Oct 30 13:33:44.629700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.629622    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-plugins-config-volume\") pod \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") "
Oct 30 13:33:44.629765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.629704    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-routing-overlay-volume\") pod \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\" (UID: \"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6\") "
Oct 30 13:33:44.630087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.630057    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-routing-overlay-volume" pod "ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" (UID: "ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:44.630087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.630068    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-plugins-config-volume" pod "ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" (UID: "ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:33:44.630495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.630388    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:33:44.630495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.630405    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:33:44.632014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.631985    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-kube-api-access-j8fjp" pod "ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" (UID: "ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6"). InnerVolumeSpecName "kube-api-access-j8fjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:44.632145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.632047    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-provider-credentials" pod "ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" (UID: "ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:44.710746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.710707    4429 generic.go:356] "Generic (PLEG): container finished" podID="ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" containerID="363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462" exitCode=0
Oct 30 13:33:44.711134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.710760    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"
Oct 30 13:33:44.711134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.710795    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm" event={"ID":"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6","Type":"ContainerDied","Data":"363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462"}
Oct 30 13:33:44.711134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.710845    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm" event={"ID":"ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6","Type":"ContainerDied","Data":"34053a4bf51e28de7d7423610aa12afa466616783f376a1d752853b4c1494b0e"}
Oct 30 13:33:44.711134 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.710868    4429 scope.go:122] "RemoveContainer" containerID="363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462"
Oct 30 13:33:44.722771 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.722747    4429 scope.go:122] "RemoveContainer" containerID="363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462"
Oct 30 13:33:44.723116 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:33:44.723091    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462\": container with ID starting with 363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462 not found: ID does not exist" containerID="363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462"
Oct 30 13:33:44.723174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.723126    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462"} err="failed to get container status \"363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462\": rpc error: code = NotFound desc = could not find container \"363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462\": container with ID starting with 363504ac2f82f24832474d9a6de593f15835e7f32762f0d0448fff54018a7462 not found: ID does not exist"
Oct 30 13:33:44.731610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.731582    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:33:44.731610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.731606    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8fjp\" (UniqueName: \"kubernetes.io/projected/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6-kube-api-access-j8fjp\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:33:44.775948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.775917    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"]
Oct 30 13:33:44.779678 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:44.779652    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-acf9e2/payload-processing-external-model-e2e-extmod-a-acf9e2-64866zqrm"]
Oct 30 13:33:46.078121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:46.078086    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" path="/var/lib/kubelet/pods/ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6/volumes"
Oct 30 13:33:52.856808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.856778    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"]
Oct 30 13:33:52.857571 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.857556    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" containerName="payload-processing"
Oct 30 13:33:52.857625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.857572    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" containerName="payload-processing"
Oct 30 13:33:52.857708 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.857698    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae856cfe-bcfd-4cc5-b902-54a85b7cb9d6" containerName="payload-processing"
Oct 30 13:33:52.863075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.863057    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.884386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.884360    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0d9115-openshift-default-dockercfg-7m5k5\""
Oct 30 13:33:52.904547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.904514    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"]
Oct 30 13:33:52.933038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.933000    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-envoy\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.933357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.933332    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.933471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.933458    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-certs\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.933568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.933555    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istiod-ca-cert\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.933746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.933730    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8rk\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-kube-api-access-kz8rk\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.933871 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.933858    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-token\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.934005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.933992    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-data\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.934097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.934085    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-credential-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:52.934189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:52.934177    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-podinfo\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.035984    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036040    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-certs\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036063    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istiod-ca-cert\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036408    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8rk\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-kube-api-access-kz8rk\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036493 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036472    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-token\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036552    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-data\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036623 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036587    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-credential-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036619    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-podinfo\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036625    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-certs\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.036806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.036787    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-envoy\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.037249 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.037221    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.037453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.037237    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-data\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.037453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.037267    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-credential-socket\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.037900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.037873    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istiod-ca-cert\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.039870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.039852    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-podinfo\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.040272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.040250    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-envoy\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.063577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.063558    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8rk\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-kube-api-access-kz8rk\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.068818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.068799    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-token\") pod \"e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") " pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.180145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.180093    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:53.524940 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:33:53.524904    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d53a6d_0f35_4bb9_b1ec_77df3434474a.slice/crio-fb5180a5fb605da2a1b1d98b04e8b87acce30bcf5c970f9cd1506206af7f9685 WatchSource:0}: Error finding container fb5180a5fb605da2a1b1d98b04e8b87acce30bcf5c970f9cd1506206af7f9685: Status 404 returned error can't find the container with id fb5180a5fb605da2a1b1d98b04e8b87acce30bcf5c970f9cd1506206af7f9685
Oct 30 13:33:53.526053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.526031    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"]
Oct 30 13:33:53.527957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.527928    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:53.528038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.527992    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:53.528038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.528022    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:53.825847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.825747    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" event={"ID":"d4d53a6d-0f35-4bb9-b1ec-77df3434474a","Type":"ContainerStarted","Data":"b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882"}
Oct 30 13:33:53.825847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.825816    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" event={"ID":"d4d53a6d-0f35-4bb9-b1ec-77df3434474a","Type":"ContainerStarted","Data":"fb5180a5fb605da2a1b1d98b04e8b87acce30bcf5c970f9cd1506206af7f9685"}
Oct 30 13:33:53.877915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:53.877852    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" podStartSLOduration=1.877808605 podStartE2EDuration="1.877808605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:53.875773408 +0000 UTC m=+3691.931267233" watchObservedRunningTime="2026-10-30 13:33:53.877808605 +0000 UTC m=+3691.933302430"
Oct 30 13:33:54.180677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:54.180590    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:33:54.182639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:54.182611    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:33:54.182775 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:54.182663    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:33:55.181039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:55.181006    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:33:55.181449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:55.181055    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:33:56.180778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:56.180690    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:33:56.180778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:56.180751    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:33:57.181021 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:57.180981    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:33:57.181537 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:57.181042    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:33:58.180595 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:58.180557    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:33:58.180823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:58.180623    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:33:59.181502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:59.181466    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 13:33:59.181933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:33:59.181533    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 13:34:00.185729 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:00.185700    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:34:00.186197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:00.186180    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:34:00.186828 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:00.186813    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:34:12.379042 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:34:12.378996    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:34:42.372470 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:34:42.372397    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:34:56.468042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.468004    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"]
Oct 30 13:34:56.473434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.473407    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.497401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.497369    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"]
Oct 30 13:34:56.521802 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.521759    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.521999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.521936    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.522159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.522139    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.522272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.522251    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9sr\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-kube-api-access-fs9sr\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.624013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.623975    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.624207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.624117    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.624207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.624169    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9sr\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-kube-api-access-fs9sr\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.624350 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.624335    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.624989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.624905    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.624989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.624945    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.626799 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.626778    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.693570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.693533    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9sr\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-kube-api-access-fs9sr\") pod \"payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") " pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:56.790217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:56.790100    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:57.099069 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:34:57.099037    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0352e27_158e_4c2c_9e23_f3c34b9b47e7.slice/crio-efe16ebacbea875bad023bb225bb7fff8f7eb9dbdebe0e40ee68c3a9fba8fa3b WatchSource:0}: Error finding container efe16ebacbea875bad023bb225bb7fff8f7eb9dbdebe0e40ee68c3a9fba8fa3b: Status 404 returned error can't find the container with id efe16ebacbea875bad023bb225bb7fff8f7eb9dbdebe0e40ee68c3a9fba8fa3b
Oct 30 13:34:57.099638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.099615    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"]
Oct 30 13:34:57.374558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.374456    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2" event={"ID":"c0352e27-158e-4c2c-9e23-f3c34b9b47e7","Type":"ContainerStarted","Data":"8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0"}
Oct 30 13:34:57.374558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.374512    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2" event={"ID":"c0352e27-158e-4c2c-9e23-f3c34b9b47e7","Type":"ContainerStarted","Data":"efe16ebacbea875bad023bb225bb7fff8f7eb9dbdebe0e40ee68c3a9fba8fa3b"}
Oct 30 13:34:57.374558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.374545    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:34:57.414689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.414653    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"]
Oct 30 13:34:57.419505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.419484    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.427489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.427091    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2" podStartSLOduration=1.427069835 podStartE2EDuration="1.427069835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:57.42570841 +0000 UTC m=+3755.481202259" watchObservedRunningTime="2026-10-30 13:34:57.427069835 +0000 UTC m=+3755.482563664"
Oct 30 13:34:57.444028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.443989    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"]
Oct 30 13:34:57.539670 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.539629    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwhh\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-kube-api-access-nwwhh\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.540143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.539784    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.540143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.539944    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.540143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.540016    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.641658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.641569    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwhh\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-kube-api-access-nwwhh\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.641840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.641657    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.641840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.641758    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.641840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.641802    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.642683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.642651    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.642815 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.642727    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.644491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.644463    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.669612 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.669573    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwhh\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-kube-api-access-nwwhh\") pod \"payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") " pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:57.739655 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:57.739623    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:58.054581 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:34:58.054548    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798a8332_0d82_473f_8f1c_0c858dbe4c1b.slice/crio-f185c9665f1f1cba9411f88217cbc9d1d9eb72dea96f879c9b0747badb420666 WatchSource:0}: Error finding container f185c9665f1f1cba9411f88217cbc9d1d9eb72dea96f879c9b0747badb420666: Status 404 returned error can't find the container with id f185c9665f1f1cba9411f88217cbc9d1d9eb72dea96f879c9b0747badb420666
Oct 30 13:34:58.055844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:58.055820    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"]
Oct 30 13:34:58.384687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:58.384581    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2" event={"ID":"798a8332-0d82-473f-8f1c-0c858dbe4c1b","Type":"ContainerStarted","Data":"89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a"}
Oct 30 13:34:58.384687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:58.384641    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2" event={"ID":"798a8332-0d82-473f-8f1c-0c858dbe4c1b","Type":"ContainerStarted","Data":"f185c9665f1f1cba9411f88217cbc9d1d9eb72dea96f879c9b0747badb420666"}
Oct 30 13:34:58.384936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:58.384686    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:34:58.433893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:34:58.433837    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2" podStartSLOduration=1.433821159 podStartE2EDuration="1.433821159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:58.431029013 +0000 UTC m=+3756.486522837" watchObservedRunningTime="2026-10-30 13:34:58.433821159 +0000 UTC m=+3756.489315028"
Oct 30 13:35:05.148302 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.148256    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"]
Oct 30 13:35:05.153677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.153644    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.178300 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.178255    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"]
Oct 30 13:35:05.185654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.185621    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-22efcf-serving-cert\""
Oct 30 13:35:05.277911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.277871    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-metrics-certs\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.278102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.277932    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c241fb9d-2a90-474e-be70-2abca4252e27-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.278207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.278182    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4sq\" (UniqueName: \"kubernetes.io/projected/c241fb9d-2a90-474e-be70-2abca4252e27-kube-api-access-7w4sq\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.278336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.278316    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-maas-api-tls\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.380367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.380332    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4sq\" (UniqueName: \"kubernetes.io/projected/c241fb9d-2a90-474e-be70-2abca4252e27-kube-api-access-7w4sq\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.380568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.380408    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-maas-api-tls\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.380568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.380491    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-metrics-certs\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.380568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.380532    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c241fb9d-2a90-474e-be70-2abca4252e27-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.381264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.381240    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c241fb9d-2a90-474e-be70-2abca4252e27-openshift-service-ca\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.383301 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.383267    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-metrics-certs\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.383407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.383369    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-maas-api-tls\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.415777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.407128    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4sq\" (UniqueName: \"kubernetes.io/projected/c241fb9d-2a90-474e-be70-2abca4252e27-kube-api-access-7w4sq\") pod \"maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.469639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.469602    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:05.784441 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:35:05.784402    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc241fb9d_2a90_474e_be70_2abca4252e27.slice/crio-9957ef82084f6a1ee38986d08242f5254d554e1d3ca5c3cb53a00c47b4e8b6c7 WatchSource:0}: Error finding container 9957ef82084f6a1ee38986d08242f5254d554e1d3ca5c3cb53a00c47b4e8b6c7: Status 404 returned error can't find the container with id 9957ef82084f6a1ee38986d08242f5254d554e1d3ca5c3cb53a00c47b4e8b6c7
Oct 30 13:35:05.785971 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:05.785945    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"]
Oct 30 13:35:06.493332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:06.493268    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z" event={"ID":"c241fb9d-2a90-474e-be70-2abca4252e27","Type":"ContainerStarted","Data":"645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c"}
Oct 30 13:35:06.493864 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:06.493368    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z" event={"ID":"c241fb9d-2a90-474e-be70-2abca4252e27","Type":"ContainerStarted","Data":"9957ef82084f6a1ee38986d08242f5254d554e1d3ca5c3cb53a00c47b4e8b6c7"}
Oct 30 13:35:06.495453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:06.494670    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:06.543461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:06.543339    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z" podStartSLOduration=1.074936179 podStartE2EDuration="1.543324161s" totalImagesPullingTime="468.387982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:06.540716514 +0000 UTC m=+3764.596210338" watchObservedRunningTime="2026-10-30 13:35:06.543324161 +0000 UTC m=+3764.598817985"
Oct 30 13:35:08.389069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:08.389032    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:35:08.470503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:08.470464    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"]
Oct 30 13:35:09.395868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:09.395835    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:35:09.472667 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:09.472622    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"]
Oct 30 13:35:12.378488 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:12.378450    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:35:13.475446 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.475144    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b" podUID="6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" containerName="payload-processing" containerID="cri-o://c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79" gracePeriod=25
Oct 30 13:35:13.525568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.525544    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:35:13.782376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.782350    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:35:13.815404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.815374    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-kube-api-access-n5f86\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-kube-api-access-n5f86\") pod \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") "
Oct 30 13:35:13.815619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.815419    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-routing-overlay-volume\") pod \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") "
Oct 30 13:35:13.815619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.815483    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-provider-credentials\") pod \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") "
Oct 30 13:35:13.815619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.815587    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-plugins-config-volume\") pod \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\" (UID: \"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247\") "
Oct 30 13:35:13.815927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.815895    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-routing-overlay-volume" pod "6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" (UID: "6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:13.816039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.816017    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-plugins-config-volume" pod "6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" (UID: "6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:13.816301 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.816246    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:13.816301 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.816268    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:13.817981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.817958    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-provider-credentials" pod "6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" (UID: "6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:13.818085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.817990    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-kube-api-access-n5f86" pod "6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" (UID: "6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247"). InnerVolumeSpecName "kube-api-access-n5f86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:13.918032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.917989    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5f86\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-kube-api-access-n5f86\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:13.918032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:13.918034    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:14.473739 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.473680    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx" podUID="c23fd937-ce25-4779-9cb8-d55e6486a656" containerName="payload-processing" containerID="cri-o://f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f" gracePeriod=25
Oct 30 13:35:14.565911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.565877    4429 generic.go:356] "Generic (PLEG): container finished" podID="6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" containerID="c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79" exitCode=0
Oct 30 13:35:14.566421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.565931    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"
Oct 30 13:35:14.566421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.565968    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b" event={"ID":"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247","Type":"ContainerDied","Data":"c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79"}
Oct 30 13:35:14.566421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.566033    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b" event={"ID":"6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247","Type":"ContainerDied","Data":"7b67718b54b7634d2b9403f684704b2b8b5a6678862fcf547aa211aba23c29a7"}
Oct 30 13:35:14.566421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.566056    4429 scope.go:122] "RemoveContainer" containerID="c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79"
Oct 30 13:35:14.588569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.588546    4429 scope.go:122] "RemoveContainer" containerID="c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79"
Oct 30 13:35:14.588890 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:14.588870    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79\": container with ID starting with c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79 not found: ID does not exist" containerID="c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79"
Oct 30 13:35:14.589032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.588897    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79"} err="failed to get container status \"c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79\": rpc error: code = NotFound desc = could not find container \"c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79\": container with ID starting with c513d24bae1c7774d4ccf8270187aca9829639f1dfc9e41e851f6d93e1c7df79 not found: ID does not exist"
Oct 30 13:35:14.626926 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.626888    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"]
Oct 30 13:35:14.640470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.640433    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-ffd6w5z7b"]
Oct 30 13:35:14.778364 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.778338    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:35:14.833024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.832992    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-provider-credentials\") pod \"c23fd937-ce25-4779-9cb8-d55e6486a656\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") "
Oct 30 13:35:14.833227 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.833038    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-routing-overlay-volume\") pod \"c23fd937-ce25-4779-9cb8-d55e6486a656\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") "
Oct 30 13:35:14.833227 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.833101    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-kube-api-access-wtqmx\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-kube-api-access-wtqmx\") pod \"c23fd937-ce25-4779-9cb8-d55e6486a656\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") "
Oct 30 13:35:14.833399 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.833246    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-plugins-config-volume\") pod \"c23fd937-ce25-4779-9cb8-d55e6486a656\" (UID: \"c23fd937-ce25-4779-9cb8-d55e6486a656\") "
Oct 30 13:35:14.833588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.833563    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-routing-overlay-volume" pod "c23fd937-ce25-4779-9cb8-d55e6486a656" (UID: "c23fd937-ce25-4779-9cb8-d55e6486a656"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:14.834064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.833923    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-plugins-config-volume" pod "c23fd937-ce25-4779-9cb8-d55e6486a656" (UID: "c23fd937-ce25-4779-9cb8-d55e6486a656"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:14.834169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.834037    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:14.835842 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.835811    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-kube-api-access-wtqmx" pod "c23fd937-ce25-4779-9cb8-d55e6486a656" (UID: "c23fd937-ce25-4779-9cb8-d55e6486a656"). InnerVolumeSpecName "kube-api-access-wtqmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:14.835944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.835899    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-provider-credentials" pod "c23fd937-ce25-4779-9cb8-d55e6486a656" (UID: "c23fd937-ce25-4779-9cb8-d55e6486a656"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:14.936068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.936029    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:14.936068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.936065    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtqmx\" (UniqueName: \"kubernetes.io/projected/c23fd937-ce25-4779-9cb8-d55e6486a656-kube-api-access-wtqmx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:14.936068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:14.936074    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23fd937-ce25-4779-9cb8-d55e6486a656-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.576929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.576887    4429 generic.go:356] "Generic (PLEG): container finished" podID="c23fd937-ce25-4779-9cb8-d55e6486a656" containerID="f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f" exitCode=0
Oct 30 13:35:15.576929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.576938    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"
Oct 30 13:35:15.577590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.576982    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx" event={"ID":"c23fd937-ce25-4779-9cb8-d55e6486a656","Type":"ContainerDied","Data":"f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f"}
Oct 30 13:35:15.577590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.577050    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx" event={"ID":"c23fd937-ce25-4779-9cb8-d55e6486a656","Type":"ContainerDied","Data":"5309cdf4fa43bb80fa9b095c752b31b48f8b266d4d53cf5a6b78fdf2ede85590"}
Oct 30 13:35:15.577590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.577098    4429 scope.go:122] "RemoveContainer" containerID="f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f"
Oct 30 13:35:15.590713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.590693    4429 scope.go:122] "RemoveContainer" containerID="f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f"
Oct 30 13:35:15.591041 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:15.591015    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f\": container with ID starting with f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f not found: ID does not exist" containerID="f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f"
Oct 30 13:35:15.591147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.591050    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f"} err="failed to get container status \"f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f\": rpc error: code = NotFound desc = could not find container \"f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f\": container with ID starting with f6cd0aae91899f4b20e543e6c71f5e5b9bd594db17d5b49e53a25a1b422d151f not found: ID does not exist"
Oct 30 13:35:15.638741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.638705    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"]
Oct 30 13:35:15.651481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:15.651451    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-789ddlxkx"]
Oct 30 13:35:16.083352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:16.083274    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" path="/var/lib/kubelet/pods/6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247/volumes"
Oct 30 13:35:16.084098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:16.084072    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23fd937-ce25-4779-9cb8-d55e6486a656" path="/var/lib/kubelet/pods/c23fd937-ce25-4779-9cb8-d55e6486a656/volumes"
Oct 30 13:35:19.051328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:19.051279    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"]
Oct 30 13:35:19.966580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:19.966546    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"]
Oct 30 13:35:23.495681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.495645    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"]
Oct 30 13:35:23.496503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.496488    4429 cpu_manager.go:414] "containerMap: removing container" podUID="6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" containerName="payload-processing"
Oct 30 13:35:23.496576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.496505    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" containerName="payload-processing"
Oct 30 13:35:23.496576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.496570    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c23fd937-ce25-4779-9cb8-d55e6486a656" containerName="payload-processing"
Oct 30 13:35:23.496576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.496575    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23fd937-ce25-4779-9cb8-d55e6486a656" containerName="payload-processing"
Oct 30 13:35:23.496699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.496689    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23fd937-ce25-4779-9cb8-d55e6486a656" containerName="payload-processing"
Oct 30 13:35:23.496735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.496705    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc9a7f5-4bed-4e75-ac09-d99b8c2b9247" containerName="payload-processing"
Oct 30 13:35:23.502262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.502239    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.512697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.512674    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gqkm\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-kube-api-access-5gqkm\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.512851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.512711    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.513102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.513079    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.513190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.513123    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.548437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.548397    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"routing-overlay\""
Oct 30 13:35:23.549041 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.549013    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-e2e--e20aaa17-dockercfg-c4rvp\""
Oct 30 13:35:23.549180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.549132    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"openshift-service-ca.crt\""
Oct 30 13:35:23.549231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.549202    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"agc-cred-d3093dc79fb2\""
Oct 30 13:35:23.549341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.549326    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2d2a0b\""
Oct 30 13:35:23.549467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.549453    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2d2a0b\"/\"kube-root-ca.crt\""
Oct 30 13:35:23.556356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.556326    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"]
Oct 30 13:35:23.614273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.614235    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.614479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.614278    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.614479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.614367    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gqkm\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-kube-api-access-5gqkm\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.614479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.614409    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.615022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.614995    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.615159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.615111    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.617121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.617097    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.640429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.640395    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gqkm\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-kube-api-access-5gqkm\") pod \"payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") " pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:23.821362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:23.819485    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:24.052604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.052566    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2" podUID="798a8332-0d82-473f-8f1c-0c858dbe4c1b" containerName="payload-processing" containerID="cri-o://89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a" gracePeriod=25
Oct 30 13:35:24.216696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.194027    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"]
Oct 30 13:35:24.448252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.448225    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:35:24.561600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.561555    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-provider-credentials\") pod \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") "
Oct 30 13:35:24.562092 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.561930    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-routing-overlay-volume\") pod \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") "
Oct 30 13:35:24.562181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.562095    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-kube-api-access-nwwhh\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-kube-api-access-nwwhh\") pod \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") "
Oct 30 13:35:24.562181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.562149    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-plugins-config-volume\") pod \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\" (UID: \"798a8332-0d82-473f-8f1c-0c858dbe4c1b\") "
Oct 30 13:35:24.563219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.562689    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-routing-overlay-volume" pod "798a8332-0d82-473f-8f1c-0c858dbe4c1b" (UID: "798a8332-0d82-473f-8f1c-0c858dbe4c1b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:24.563219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.562933    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:24.563219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.563024    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-plugins-config-volume" pod "798a8332-0d82-473f-8f1c-0c858dbe4c1b" (UID: "798a8332-0d82-473f-8f1c-0c858dbe4c1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:24.564748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.564722    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-provider-credentials" pod "798a8332-0d82-473f-8f1c-0c858dbe4c1b" (UID: "798a8332-0d82-473f-8f1c-0c858dbe4c1b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:24.564900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.564769    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-kube-api-access-nwwhh" pod "798a8332-0d82-473f-8f1c-0c858dbe4c1b" (UID: "798a8332-0d82-473f-8f1c-0c858dbe4c1b"). InnerVolumeSpecName "kube-api-access-nwwhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:24.664098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.664062    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwhh\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-kube-api-access-nwwhh\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:24.664098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.664093    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798a8332-0d82-473f-8f1c-0c858dbe4c1b-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:24.664098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.664103    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/798a8332-0d82-473f-8f1c-0c858dbe4c1b-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:24.667886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.667855    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" event={"ID":"a1c2d1a9-aba7-4770-b106-68d6528e8bbe","Type":"ContainerStarted","Data":"27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a"}
Oct 30 13:35:24.668039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.667915    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" event={"ID":"a1c2d1a9-aba7-4770-b106-68d6528e8bbe","Type":"ContainerStarted","Data":"11a879e89521b3bf98ae23b5430e406349254abc32248a512f2853c7ba64e899"}
Oct 30 13:35:24.668039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.668011    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:24.669466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.669442    4429 generic.go:356] "Generic (PLEG): container finished" podID="798a8332-0d82-473f-8f1c-0c858dbe4c1b" containerID="89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a" exitCode=0
Oct 30 13:35:24.669569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.669486    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"
Oct 30 13:35:24.669569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.669517    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2" event={"ID":"798a8332-0d82-473f-8f1c-0c858dbe4c1b","Type":"ContainerDied","Data":"89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a"}
Oct 30 13:35:24.669569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.669564    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2" event={"ID":"798a8332-0d82-473f-8f1c-0c858dbe4c1b","Type":"ContainerDied","Data":"f185c9665f1f1cba9411f88217cbc9d1d9eb72dea96f879c9b0747badb420666"}
Oct 30 13:35:24.669714 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.669588    4429 scope.go:122] "RemoveContainer" containerID="89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a"
Oct 30 13:35:24.682012 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.681991    4429 scope.go:122] "RemoveContainer" containerID="89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a"
Oct 30 13:35:24.682381 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:24.682333    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a\": container with ID starting with 89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a not found: ID does not exist" containerID="89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a"
Oct 30 13:35:24.682381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.682363    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a"} err="failed to get container status \"89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a\": rpc error: code = NotFound desc = could not find container \"89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a\": container with ID starting with 89345246bded12f168008900d245d00574d33d1e66ef8048bd9df9b1d864a98a not found: ID does not exist"
Oct 30 13:35:24.723359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.723265    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" podStartSLOduration=1.7232508050000002 podStartE2EDuration="1.723250805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:24.720861907 +0000 UTC m=+3782.776355759" watchObservedRunningTime="2026-10-30 13:35:24.723250805 +0000 UTC m=+3782.778744630"
Oct 30 13:35:24.767381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.767344    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"]
Oct 30 13:35:24.771180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.771153    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4db150/payload-processing-external-model-e2e-extmod-a-4db150-848ctvdp2"]
Oct 30 13:35:24.967514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:24.967459    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2" podUID="c0352e27-158e-4c2c-9e23-f3c34b9b47e7" containerName="payload-processing" containerID="cri-o://8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0" gracePeriod=25
Oct 30 13:35:25.255130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.255102    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:35:25.372826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.372795    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-provider-credentials\") pod \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") "
Oct 30 13:35:25.373015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.372859    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-routing-overlay-volume\") pod \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") "
Oct 30 13:35:25.373015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.372901    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-kube-api-access-fs9sr\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-kube-api-access-fs9sr\") pod \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") "
Oct 30 13:35:25.373015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.372923    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-plugins-config-volume\") pod \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\" (UID: \"c0352e27-158e-4c2c-9e23-f3c34b9b47e7\") "
Oct 30 13:35:25.373385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.373356    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-routing-overlay-volume" pod "c0352e27-158e-4c2c-9e23-f3c34b9b47e7" (UID: "c0352e27-158e-4c2c-9e23-f3c34b9b47e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:25.373519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.373445    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-plugins-config-volume" pod "c0352e27-158e-4c2c-9e23-f3c34b9b47e7" (UID: "c0352e27-158e-4c2c-9e23-f3c34b9b47e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:25.373592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.373579    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:25.373631 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.373595    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:25.375235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.375214    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-provider-credentials" pod "c0352e27-158e-4c2c-9e23-f3c34b9b47e7" (UID: "c0352e27-158e-4c2c-9e23-f3c34b9b47e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:25.375387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.375241    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-kube-api-access-fs9sr" pod "c0352e27-158e-4c2c-9e23-f3c34b9b47e7" (UID: "c0352e27-158e-4c2c-9e23-f3c34b9b47e7"). InnerVolumeSpecName "kube-api-access-fs9sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:25.475088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.475013    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:25.475088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.475043    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs9sr\" (UniqueName: \"kubernetes.io/projected/c0352e27-158e-4c2c-9e23-f3c34b9b47e7-kube-api-access-fs9sr\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:25.680673 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.680588    4429 generic.go:356] "Generic (PLEG): container finished" podID="c0352e27-158e-4c2c-9e23-f3c34b9b47e7" containerID="8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0" exitCode=0
Oct 30 13:35:25.680673 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.680660    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"
Oct 30 13:35:25.682061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.680737    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2" event={"ID":"c0352e27-158e-4c2c-9e23-f3c34b9b47e7","Type":"ContainerDied","Data":"8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0"}
Oct 30 13:35:25.682061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.680783    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2" event={"ID":"c0352e27-158e-4c2c-9e23-f3c34b9b47e7","Type":"ContainerDied","Data":"efe16ebacbea875bad023bb225bb7fff8f7eb9dbdebe0e40ee68c3a9fba8fa3b"}
Oct 30 13:35:25.682061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.680812    4429 scope.go:122] "RemoveContainer" containerID="8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0"
Oct 30 13:35:25.695245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.695226    4429 scope.go:122] "RemoveContainer" containerID="8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0"
Oct 30 13:35:25.695555 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:25.695529    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0\": container with ID starting with 8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0 not found: ID does not exist" containerID="8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0"
Oct 30 13:35:25.695643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.695566    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0"} err="failed to get container status \"8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0\": rpc error: code = NotFound desc = could not find container \"8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0\": container with ID starting with 8e822c78dfa207cf61bb5a52ff460e2f2ef99e8301fcd3ad3c46420a14fbdce0 not found: ID does not exist"
Oct 30 13:35:25.744153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.744094    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"]
Oct 30 13:35:25.751027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:25.751001    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-75d385/payload-processing-external-model-e2e-extmod-b-75d385-b7b5vsnp2"]
Oct 30 13:35:25.779940 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:25.779917    4429 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2d2a0b/agc-cred-d3093dc79fb2: secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:25.779940 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:25.779939    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6: secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:25.780148 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:25.780017    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials podName:a1c2d1a9-aba7-4770-b106-68d6528e8bbe nodeName:}" failed. No retries permitted until 2026-10-30 13:35:26.279997234 +0000 UTC m=+3784.335491037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" (UID: "a1c2d1a9-aba7-4770-b106-68d6528e8bbe") : secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:26.077032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:26.076960    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798a8332-0d82-473f-8f1c-0c858dbe4c1b" path="/var/lib/kubelet/pods/798a8332-0d82-473f-8f1c-0c858dbe4c1b/volumes"
Oct 30 13:35:26.077489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:26.077473    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0352e27-158e-4c2c-9e23-f3c34b9b47e7" path="/var/lib/kubelet/pods/c0352e27-158e-4c2c-9e23-f3c34b9b47e7/volumes"
Oct 30 13:35:26.288263 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:26.288228    4429 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2d2a0b/agc-cred-d3093dc79fb2: secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:26.288263 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:26.288259    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6: secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:26.288494 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:26.288358    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials podName:a1c2d1a9-aba7-4770-b106-68d6528e8bbe nodeName:}" failed. No retries permitted until 2026-10-30 13:35:27.2883373 +0000 UTC m=+3785.343831101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" (UID: "a1c2d1a9-aba7-4770-b106-68d6528e8bbe") : secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:26.952953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:26.952921    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"]
Oct 30 13:35:27.324372 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:27.311963    4429 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2d2a0b/agc-cred-d3093dc79fb2: secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:27.324372 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:27.311999    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6: secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:27.324372 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:27.312064    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials podName:a1c2d1a9-aba7-4770-b106-68d6528e8bbe nodeName:}" failed. No retries permitted until 2026-10-30 13:35:29.312044204 +0000 UTC m=+3787.367538005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" (UID: "a1c2d1a9-aba7-4770-b106-68d6528e8bbe") : secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:29.352306 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:29.352258    4429 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2d2a0b/agc-cred-d3093dc79fb2: secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:29.352306 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:29.352298    4429 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6: secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:29.352920 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:29.352368    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials podName:a1c2d1a9-aba7-4770-b106-68d6528e8bbe nodeName:}" failed. No retries permitted until 2026-10-30 13:35:33.352347374 +0000 UTC m=+3791.407841180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" (UID: "a1c2d1a9-aba7-4770-b106-68d6528e8bbe") : secret "agc-cred-d3093dc79fb2" not found
Oct 30 13:35:31.953772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:31.953733    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" podUID="a1c2d1a9-aba7-4770-b106-68d6528e8bbe" containerName="payload-processing" containerID="cri-o://27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a" gracePeriod=25
Oct 30 13:35:32.235038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.235014    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:32.401566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.401533    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-plugins-config-volume\") pod \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") "
Oct 30 13:35:32.401756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.401642    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-kube-api-access-5gqkm\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-kube-api-access-5gqkm\") pod \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") "
Oct 30 13:35:32.401756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.401747    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials\") pod \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") "
Oct 30 13:35:32.401874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.401821    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-routing-overlay-volume\") pod \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\" (UID: \"a1c2d1a9-aba7-4770-b106-68d6528e8bbe\") "
Oct 30 13:35:32.402013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.401989    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-plugins-config-volume" pod "a1c2d1a9-aba7-4770-b106-68d6528e8bbe" (UID: "a1c2d1a9-aba7-4770-b106-68d6528e8bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:32.402204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.402174    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-routing-overlay-volume" pod "a1c2d1a9-aba7-4770-b106-68d6528e8bbe" (UID: "a1c2d1a9-aba7-4770-b106-68d6528e8bbe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:32.402332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.402224    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:32.404026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.404005    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-kube-api-access-5gqkm" pod "a1c2d1a9-aba7-4770-b106-68d6528e8bbe" (UID: "a1c2d1a9-aba7-4770-b106-68d6528e8bbe"). InnerVolumeSpecName "kube-api-access-5gqkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:32.404120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.404036    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials" pod "a1c2d1a9-aba7-4770-b106-68d6528e8bbe" (UID: "a1c2d1a9-aba7-4770-b106-68d6528e8bbe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:32.512362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.507583    4429 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-routing-overlay-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:32.512362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.507620    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gqkm\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-kube-api-access-5gqkm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:32.512362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.507636    4429 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1c2d1a9-aba7-4770-b106-68d6528e8bbe-provider-credentials\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:35:32.735023 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.734988    4429 generic.go:356] "Generic (PLEG): container finished" podID="a1c2d1a9-aba7-4770-b106-68d6528e8bbe" containerID="27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a" exitCode=0
Oct 30 13:35:32.735241 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.735037    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"
Oct 30 13:35:32.735241 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.735079    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" event={"ID":"a1c2d1a9-aba7-4770-b106-68d6528e8bbe","Type":"ContainerDied","Data":"27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a"}
Oct 30 13:35:32.735241 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.735142    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6" event={"ID":"a1c2d1a9-aba7-4770-b106-68d6528e8bbe","Type":"ContainerDied","Data":"11a879e89521b3bf98ae23b5430e406349254abc32248a512f2853c7ba64e899"}
Oct 30 13:35:32.735241 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.735175    4429 scope.go:122] "RemoveContainer" containerID="27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a"
Oct 30 13:35:32.746925 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.746903    4429 scope.go:122] "RemoveContainer" containerID="27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a"
Oct 30 13:35:32.747202 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:32.747184    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a\": container with ID starting with 27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a not found: ID does not exist" containerID="27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a"
Oct 30 13:35:32.747272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.747210    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a"} err="failed to get container status \"27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a\": rpc error: code = NotFound desc = could not find container \"27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a\": container with ID starting with 27c7fa9321fef4f452cb345b54f441d64cb7628bc0c90bd6e3eab4cab8dc4a1a not found: ID does not exist"
Oct 30 13:35:32.796560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.796502    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"]
Oct 30 13:35:32.810413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:32.810389    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2d2a0b/payload-processing-external-model-e2e-extmod-b-2d2a0b-6c89jxdp6"]
Oct 30 13:35:34.077780 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.077750    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c2d1a9-aba7-4770-b106-68d6528e8bbe" path="/var/lib/kubelet/pods/a1c2d1a9-aba7-4770-b106-68d6528e8bbe/volumes"
Oct 30 13:35:34.216737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.216699    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"]
Oct 30 13:35:34.217514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217499    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a1c2d1a9-aba7-4770-b106-68d6528e8bbe" containerName="payload-processing"
Oct 30 13:35:34.217573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217516    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c2d1a9-aba7-4770-b106-68d6528e8bbe" containerName="payload-processing"
Oct 30 13:35:34.217573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217535    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c0352e27-158e-4c2c-9e23-f3c34b9b47e7" containerName="payload-processing"
Oct 30 13:35:34.217573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217540    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0352e27-158e-4c2c-9e23-f3c34b9b47e7" containerName="payload-processing"
Oct 30 13:35:34.217573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217565    4429 cpu_manager.go:414] "containerMap: removing container" podUID="798a8332-0d82-473f-8f1c-0c858dbe4c1b" containerName="payload-processing"
Oct 30 13:35:34.217573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217572    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798a8332-0d82-473f-8f1c-0c858dbe4c1b" containerName="payload-processing"
Oct 30 13:35:34.217747 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217694    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c2d1a9-aba7-4770-b106-68d6528e8bbe" containerName="payload-processing"
Oct 30 13:35:34.217747 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217706    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0352e27-158e-4c2c-9e23-f3c34b9b47e7" containerName="payload-processing"
Oct 30 13:35:34.217747 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.217716    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="798a8332-0d82-473f-8f1c-0c858dbe4c1b" containerName="payload-processing"
Oct 30 13:35:34.222817 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.222798    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:34.242020 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.242000    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"]
Oct 30 13:35:34.250856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.250833    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-025c7f\""
Oct 30 13:35:34.250998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.250977    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-025c7f-dockercfg-5bcrx\""
Oct 30 13:35:34.292469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.290204    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"]
Oct 30 13:35:34.324600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.303650    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:34.346242 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.346159    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719a6fc2-d535-4266-b8c7-52c20254c55f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq\" (UID: \"719a6fc2-d535-4266-b8c7-52c20254c55f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:34.383180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.383124    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76vv\" (UniqueName: \"kubernetes.io/projected/719a6fc2-d535-4266-b8c7-52c20254c55f-kube-api-access-p76vv\") pod \"payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq\" (UID: \"719a6fc2-d535-4266-b8c7-52c20254c55f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:34.385910 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.385876    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"]
Oct 30 13:35:34.485412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.485377    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9jf\" (UniqueName: \"kubernetes.io/projected/8cf78b02-2d46-480c-9b35-0511616757cf-kube-api-access-sm9jf\") pod \"payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf\" (UID: \"8cf78b02-2d46-480c-9b35-0511616757cf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:34.485599 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.485436    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76vv\" (UniqueName: \"kubernetes.io/projected/719a6fc2-d535-4266-b8c7-52c20254c55f-kube-api-access-p76vv\") pod \"payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq\" (UID: \"719a6fc2-d535-4266-b8c7-52c20254c55f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:34.485718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.485693    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719a6fc2-d535-4266-b8c7-52c20254c55f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq\" (UID: \"719a6fc2-d535-4266-b8c7-52c20254c55f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:34.485805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.485789    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf78b02-2d46-480c-9b35-0511616757cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf\" (UID: \"8cf78b02-2d46-480c-9b35-0511616757cf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:34.486477 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.486458    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719a6fc2-d535-4266-b8c7-52c20254c55f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq\" (UID: \"719a6fc2-d535-4266-b8c7-52c20254c55f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:34.511369 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.511334    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76vv\" (UniqueName: \"kubernetes.io/projected/719a6fc2-d535-4266-b8c7-52c20254c55f-kube-api-access-p76vv\") pod \"payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq\" (UID: \"719a6fc2-d535-4266-b8c7-52c20254c55f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:34.537559 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.537531    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:34.587310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.587247    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf78b02-2d46-480c-9b35-0511616757cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf\" (UID: \"8cf78b02-2d46-480c-9b35-0511616757cf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:34.587467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.587326    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9jf\" (UniqueName: \"kubernetes.io/projected/8cf78b02-2d46-480c-9b35-0511616757cf-kube-api-access-sm9jf\") pod \"payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf\" (UID: \"8cf78b02-2d46-480c-9b35-0511616757cf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:34.588177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.588153    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf78b02-2d46-480c-9b35-0511616757cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf\" (UID: \"8cf78b02-2d46-480c-9b35-0511616757cf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:34.612689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.612656    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9jf\" (UniqueName: \"kubernetes.io/projected/8cf78b02-2d46-480c-9b35-0511616757cf-kube-api-access-sm9jf\") pod \"payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf\" (UID: \"8cf78b02-2d46-480c-9b35-0511616757cf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:34.686849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.686818    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:34.857186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:34.857156    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"]
Oct 30 13:35:34.857787 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:35:34.857760    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719a6fc2_d535_4266_b8c7_52c20254c55f.slice/crio-d21b49b34e0a73175850985bcf6df37e3c70b84621d423d2304d40610a628e7e WatchSource:0}: Error finding container d21b49b34e0a73175850985bcf6df37e3c70b84621d423d2304d40610a628e7e: Status 404 returned error can't find the container with id d21b49b34e0a73175850985bcf6df37e3c70b84621d423d2304d40610a628e7e
Oct 30 13:35:35.020932 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:35:35.020901    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf78b02_2d46_480c_9b35_0511616757cf.slice/crio-e2fee697005f089168a11fd339ca4f5195e216c720dff0cd59a0cfa9adeddd98 WatchSource:0}: Error finding container e2fee697005f089168a11fd339ca4f5195e216c720dff0cd59a0cfa9adeddd98: Status 404 returned error can't find the container with id e2fee697005f089168a11fd339ca4f5195e216c720dff0cd59a0cfa9adeddd98
Oct 30 13:35:35.022810 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.022787    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"]
Oct 30 13:35:35.764230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.764196    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf" event={"ID":"8cf78b02-2d46-480c-9b35-0511616757cf","Type":"ContainerStarted","Data":"6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba"}
Oct 30 13:35:35.764738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.764255    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf" event={"ID":"8cf78b02-2d46-480c-9b35-0511616757cf","Type":"ContainerStarted","Data":"e2fee697005f089168a11fd339ca4f5195e216c720dff0cd59a0cfa9adeddd98"}
Oct 30 13:35:35.764738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.764315    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:35.765981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.765949    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq" event={"ID":"719a6fc2-d535-4266-b8c7-52c20254c55f","Type":"ContainerStarted","Data":"53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734"}
Oct 30 13:35:35.766094 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.765996    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq" event={"ID":"719a6fc2-d535-4266-b8c7-52c20254c55f","Type":"ContainerStarted","Data":"d21b49b34e0a73175850985bcf6df37e3c70b84621d423d2304d40610a628e7e"}
Oct 30 13:35:35.766094 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.766017    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:35.835600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.835538    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf" podStartSLOduration=1.835523196 podStartE2EDuration="1.835523196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:35.833068035 +0000 UTC m=+3793.888561860" watchObservedRunningTime="2026-10-30 13:35:35.835523196 +0000 UTC m=+3793.891017020"
Oct 30 13:35:35.938305 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:35.936864    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq" podStartSLOduration=1.936844447 podStartE2EDuration="1.936844447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:35.926733586 +0000 UTC m=+3793.982227431" watchObservedRunningTime="2026-10-30 13:35:35.936844447 +0000 UTC m=+3793.992338273"
Oct 30 13:35:39.654053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.654022    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"]
Oct 30 13:35:39.659165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.659146    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:39.683449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.683425    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"]
Oct 30 13:35:39.697366 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.697338    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9a93bd-dockercfg-pvkr8\""
Oct 30 13:35:39.703209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.703188    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9a93bd\""
Oct 30 13:35:39.786066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.786033    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqd45\" (UniqueName: \"kubernetes.io/projected/1e68bd2b-8c5d-462a-a91d-c6ba92965718-kube-api-access-jqd45\") pod \"payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r\" (UID: \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:39.786241 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.786181    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68bd2b-8c5d-462a-a91d-c6ba92965718-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r\" (UID: \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:39.888309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.887511    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68bd2b-8c5d-462a-a91d-c6ba92965718-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r\" (UID: \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:39.888309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.887793    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqd45\" (UniqueName: \"kubernetes.io/projected/1e68bd2b-8c5d-462a-a91d-c6ba92965718-kube-api-access-jqd45\") pod \"payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r\" (UID: \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:39.892308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.888987    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68bd2b-8c5d-462a-a91d-c6ba92965718-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r\" (UID: \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:39.936117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.934026    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqd45\" (UniqueName: \"kubernetes.io/projected/1e68bd2b-8c5d-462a-a91d-c6ba92965718-kube-api-access-jqd45\") pod \"payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r\" (UID: \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:39.975608 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:39.975580    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:40.290571 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:35:40.290540    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e68bd2b_8c5d_462a_a91d_c6ba92965718.slice/crio-8a970fc79ee6ea94a681297f3f6583ef27f73a0e0125eea0ada5f567fb8cc3e8 WatchSource:0}: Error finding container 8a970fc79ee6ea94a681297f3f6583ef27f73a0e0125eea0ada5f567fb8cc3e8: Status 404 returned error can't find the container with id 8a970fc79ee6ea94a681297f3f6583ef27f73a0e0125eea0ada5f567fb8cc3e8
Oct 30 13:35:40.291722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:40.291699    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"]
Oct 30 13:35:40.806384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:40.806345    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r" event={"ID":"1e68bd2b-8c5d-462a-a91d-c6ba92965718","Type":"ContainerStarted","Data":"900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860"}
Oct 30 13:35:40.806798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:40.806406    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r" event={"ID":"1e68bd2b-8c5d-462a-a91d-c6ba92965718","Type":"ContainerStarted","Data":"8a970fc79ee6ea94a681297f3f6583ef27f73a0e0125eea0ada5f567fb8cc3e8"}
Oct 30 13:35:40.806798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:40.806447    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:40.854015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:40.853961    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r" podStartSLOduration=1.8539472639999999 podStartE2EDuration="1.853947264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:40.850404746 +0000 UTC m=+3798.905898569" watchObservedRunningTime="2026-10-30 13:35:40.853947264 +0000 UTC m=+3798.909441088"
Oct 30 13:35:41.480700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.480660    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"]
Oct 30 13:35:41.485684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.485663    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:41.514461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.514436    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-09ad1e\""
Oct 30 13:35:41.514581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.514477    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-09ad1e-dockercfg-9vftj\""
Oct 30 13:35:41.529054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.529027    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"]
Oct 30 13:35:41.639307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.636725    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af1c0aa-06d3-401e-94ce-021effe0ba5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz\" (UID: \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:41.639307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.637025    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd68q\" (UniqueName: \"kubernetes.io/projected/0af1c0aa-06d3-401e-94ce-021effe0ba5f-kube-api-access-cd68q\") pod \"payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz\" (UID: \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:41.641362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.641334    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"]
Oct 30 13:35:41.670718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.670686    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:41.689811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.689512    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"]
Oct 30 13:35:41.747321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.746040    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af1c0aa-06d3-401e-94ce-021effe0ba5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz\" (UID: \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:41.747321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.746148    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzx4\" (UniqueName: \"kubernetes.io/projected/2c64c968-129c-4f4e-806e-a01a1a59e267-kube-api-access-vhzx4\") pod \"payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g\" (UID: \"2c64c968-129c-4f4e-806e-a01a1a59e267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:41.749968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.749939    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c64c968-129c-4f4e-806e-a01a1a59e267-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g\" (UID: \"2c64c968-129c-4f4e-806e-a01a1a59e267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:41.750153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.750046    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd68q\" (UniqueName: \"kubernetes.io/projected/0af1c0aa-06d3-401e-94ce-021effe0ba5f-kube-api-access-cd68q\") pod \"payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz\" (UID: \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:41.750153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.750087    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af1c0aa-06d3-401e-94ce-021effe0ba5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz\" (UID: \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:41.775604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.775534    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd68q\" (UniqueName: \"kubernetes.io/projected/0af1c0aa-06d3-401e-94ce-021effe0ba5f-kube-api-access-cd68q\") pod \"payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz\" (UID: \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:41.800470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.800442    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:41.852149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.852115    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzx4\" (UniqueName: \"kubernetes.io/projected/2c64c968-129c-4f4e-806e-a01a1a59e267-kube-api-access-vhzx4\") pod \"payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g\" (UID: \"2c64c968-129c-4f4e-806e-a01a1a59e267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:41.852642 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.852331    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c64c968-129c-4f4e-806e-a01a1a59e267-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g\" (UID: \"2c64c968-129c-4f4e-806e-a01a1a59e267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:41.853078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.853060    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c64c968-129c-4f4e-806e-a01a1a59e267-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g\" (UID: \"2c64c968-129c-4f4e-806e-a01a1a59e267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:41.889668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:41.889385    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzx4\" (UniqueName: \"kubernetes.io/projected/2c64c968-129c-4f4e-806e-a01a1a59e267-kube-api-access-vhzx4\") pod \"payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g\" (UID: \"2c64c968-129c-4f4e-806e-a01a1a59e267\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:42.051614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.051531    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:42.138312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.126847    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"]
Oct 30 13:35:42.138312 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:35:42.129695    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af1c0aa_06d3_401e_94ce_021effe0ba5f.slice/crio-79ec57936361f55d1bf17d16f4fa6d21ca646259b8d110f1ce89213fe541c3ae WatchSource:0}: Error finding container 79ec57936361f55d1bf17d16f4fa6d21ca646259b8d110f1ce89213fe541c3ae: Status 404 returned error can't find the container with id 79ec57936361f55d1bf17d16f4fa6d21ca646259b8d110f1ce89213fe541c3ae
Oct 30 13:35:42.380534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.380411    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"]
Oct 30 13:35:42.413369 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:35:42.404015    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c64c968_129c_4f4e_806e_a01a1a59e267.slice/crio-838b3884fd3a4e3abe0df348f612a9799ed0e6fa7afc6cfe7c63600ca4c17b39 WatchSource:0}: Error finding container 838b3884fd3a4e3abe0df348f612a9799ed0e6fa7afc6cfe7c63600ca4c17b39: Status 404 returned error can't find the container with id 838b3884fd3a4e3abe0df348f612a9799ed0e6fa7afc6cfe7c63600ca4c17b39
Oct 30 13:35:42.418778 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:35:42.413628    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:35:42.842340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.839320    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz" event={"ID":"0af1c0aa-06d3-401e-94ce-021effe0ba5f","Type":"ContainerStarted","Data":"08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf"}
Oct 30 13:35:42.842340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.839415    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz" event={"ID":"0af1c0aa-06d3-401e-94ce-021effe0ba5f","Type":"ContainerStarted","Data":"79ec57936361f55d1bf17d16f4fa6d21ca646259b8d110f1ce89213fe541c3ae"}
Oct 30 13:35:42.842340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.840980    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:35:42.843380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.843351    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g" event={"ID":"2c64c968-129c-4f4e-806e-a01a1a59e267","Type":"ContainerStarted","Data":"8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615"}
Oct 30 13:35:42.843566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.843548    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g" event={"ID":"2c64c968-129c-4f4e-806e-a01a1a59e267","Type":"ContainerStarted","Data":"838b3884fd3a4e3abe0df348f612a9799ed0e6fa7afc6cfe7c63600ca4c17b39"}
Oct 30 13:35:42.844496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.844482    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:42.888176 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.888128    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz" podStartSLOduration=1.888112553 podStartE2EDuration="1.888112553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:42.885715123 +0000 UTC m=+3800.941208976" watchObservedRunningTime="2026-10-30 13:35:42.888112553 +0000 UTC m=+3800.943606378"
Oct 30 13:35:42.949316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:42.944975    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g" podStartSLOduration=1.9443709340000002 podStartE2EDuration="1.944370934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:42.938773786 +0000 UTC m=+3800.994267611" watchObservedRunningTime="2026-10-30 13:35:42.944370934 +0000 UTC m=+3800.999864759"
Oct 30 13:35:46.777406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:46.777371    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:35:46.777815 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:46.777454    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:35:51.821462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:51.821433    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:35:53.857888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:53.857857    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:35:54.865640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:35:54.865616    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:36:12.373501 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:36:12.373423    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:36:13.604498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.604463    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"]
Oct 30 13:36:13.610318 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.610273    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:13.627927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.627903    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"]
Oct 30 13:36:13.636390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.636358    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7b7407\""
Oct 30 13:36:13.636390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.636376    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7b7407-dockercfg-kr8m9\""
Oct 30 13:36:13.786402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.786364    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa69d05-7642-40a4-8a81-ae56b7952142-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d\" (UID: \"efa69d05-7642-40a4-8a81-ae56b7952142\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:13.822115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.822083    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjq45\" (UniqueName: \"kubernetes.io/projected/efa69d05-7642-40a4-8a81-ae56b7952142-kube-api-access-kjq45\") pod \"payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d\" (UID: \"efa69d05-7642-40a4-8a81-ae56b7952142\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:13.924526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.924435    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq45\" (UniqueName: \"kubernetes.io/projected/efa69d05-7642-40a4-8a81-ae56b7952142-kube-api-access-kjq45\") pod \"payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d\" (UID: \"efa69d05-7642-40a4-8a81-ae56b7952142\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:13.924715 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.924639    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa69d05-7642-40a4-8a81-ae56b7952142-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d\" (UID: \"efa69d05-7642-40a4-8a81-ae56b7952142\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:13.925321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:13.925301    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa69d05-7642-40a4-8a81-ae56b7952142-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d\" (UID: \"efa69d05-7642-40a4-8a81-ae56b7952142\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:14.021818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:14.021785    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjq45\" (UniqueName: \"kubernetes.io/projected/efa69d05-7642-40a4-8a81-ae56b7952142-kube-api-access-kjq45\") pod \"payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d\" (UID: \"efa69d05-7642-40a4-8a81-ae56b7952142\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:14.233008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:14.232973    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:14.542918 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:36:14.542802    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa69d05_7642_40a4_8a81_ae56b7952142.slice/crio-df6ac0e0b18752e850bdfde030b11b191433a8bebd19e0894d0bb563cc6ed8d1 WatchSource:0}: Error finding container df6ac0e0b18752e850bdfde030b11b191433a8bebd19e0894d0bb563cc6ed8d1: Status 404 returned error can't find the container with id df6ac0e0b18752e850bdfde030b11b191433a8bebd19e0894d0bb563cc6ed8d1
Oct 30 13:36:14.544596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:14.544567    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"]
Oct 30 13:36:15.113749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:15.113716    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d" event={"ID":"efa69d05-7642-40a4-8a81-ae56b7952142","Type":"ContainerStarted","Data":"445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba"}
Oct 30 13:36:15.114136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:15.113777    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:15.114136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:15.113795    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d" event={"ID":"efa69d05-7642-40a4-8a81-ae56b7952142","Type":"ContainerStarted","Data":"df6ac0e0b18752e850bdfde030b11b191433a8bebd19e0894d0bb563cc6ed8d1"}
Oct 30 13:36:15.167628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:15.167575    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d" podStartSLOduration=2.167561459 podStartE2EDuration="2.167561459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:15.165856786 +0000 UTC m=+3833.221350630" watchObservedRunningTime="2026-10-30 13:36:15.167561459 +0000 UTC m=+3833.223055319"
Oct 30 13:36:26.124676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.124647    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:36:26.715405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.715374    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"]
Oct 30 13:36:26.722382 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.722355    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:26.741655 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.741605    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-025c7f-serving-cert\""
Oct 30 13:36:26.743378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.743354    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"]
Oct 30 13:36:26.910960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.910926    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-metrics-certs\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:26.911139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.910969    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-maas-api-tls\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:26.911139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.910990    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5jm\" (UniqueName: \"kubernetes.io/projected/35af2313-f69b-4be2-ba2e-eb9987eeae3b-kube-api-access-ct5jm\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:26.911139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:26.911093    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af2313-f69b-4be2-ba2e-eb9987eeae3b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.012717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.012621    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-metrics-certs\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.012717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.012680    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-maas-api-tls\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.012717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.012713    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5jm\" (UniqueName: \"kubernetes.io/projected/35af2313-f69b-4be2-ba2e-eb9987eeae3b-kube-api-access-ct5jm\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.013018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.012775    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af2313-f69b-4be2-ba2e-eb9987eeae3b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.014118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.014030    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af2313-f69b-4be2-ba2e-eb9987eeae3b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.015647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.015625    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-metrics-certs\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.016030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.016011    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-maas-api-tls\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.044035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.044006    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5jm\" (UniqueName: \"kubernetes.io/projected/35af2313-f69b-4be2-ba2e-eb9987eeae3b-kube-api-access-ct5jm\") pod \"maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.066847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.066824    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:27.378573 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:36:27.378547    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35af2313_f69b_4be2_ba2e_eb9987eeae3b.slice/crio-1245057192ee4b243450f7744ca747e023463ef80a85bcc14b0a6ab991aa9c9f WatchSource:0}: Error finding container 1245057192ee4b243450f7744ca747e023463ef80a85bcc14b0a6ab991aa9c9f: Status 404 returned error can't find the container with id 1245057192ee4b243450f7744ca747e023463ef80a85bcc14b0a6ab991aa9c9f
Oct 30 13:36:27.380108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:27.380083    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"]
Oct 30 13:36:28.220461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:28.220423    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9" event={"ID":"35af2313-f69b-4be2-ba2e-eb9987eeae3b","Type":"ContainerStarted","Data":"0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2"}
Oct 30 13:36:28.220731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:28.220488    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9" event={"ID":"35af2313-f69b-4be2-ba2e-eb9987eeae3b","Type":"ContainerStarted","Data":"1245057192ee4b243450f7744ca747e023463ef80a85bcc14b0a6ab991aa9c9f"}
Oct 30 13:36:28.220731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:28.220518    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:28.268981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:28.268933    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9" podStartSLOduration=1.817858017 podStartE2EDuration="2.268919477s" totalImagesPullingTime="451.06146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:28.267010633 +0000 UTC m=+3846.322504457" watchObservedRunningTime="2026-10-30 13:36:28.268919477 +0000 UTC m=+3846.324413299"
Oct 30 13:36:33.673863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.673831    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"]
Oct 30 13:36:33.679784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.679764    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.701078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.701036    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"]
Oct 30 13:36:33.708305 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.708270    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9a93bd-serving-cert\""
Oct 30 13:36:33.710118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.709632    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-metrics-certs\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.710118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.709682    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.710118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.709880    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tph6w\" (UniqueName: \"kubernetes.io/projected/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-kube-api-access-tph6w\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.710118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.709998    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.811275 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.811235    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tph6w\" (UniqueName: \"kubernetes.io/projected/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-kube-api-access-tph6w\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.811519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.811396    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.811597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.811517    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-metrics-certs\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.811597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.811552    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.811758 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:36:33.811737    4429 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-serving-cert: secret "maas-api-e2e-inf-b-9a93bd-serving-cert" not found
Oct 30 13:36:33.811828 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:36:33.811817    4429 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls podName:015afa3f-7b83-4548-974f-f8f0aeb2c7ec nodeName:}" failed. No retries permitted until 2026-10-30 13:36:34.311799264 +0000 UTC m=+3852.367293069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls") pod "maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk" (UID: "015afa3f-7b83-4548-974f-f8f0aeb2c7ec") : secret "maas-api-e2e-inf-b-9a93bd-serving-cert" not found
Oct 30 13:36:33.812048 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.812027    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.813995 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.813978    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-metrics-certs\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:33.838859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:33.838835    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tph6w\" (UniqueName: \"kubernetes.io/projected/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-kube-api-access-tph6w\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:34.233784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:34.233756    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:36:34.317978 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:34.317937    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:34.320631 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:34.320602    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:34.596201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:34.596123    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:34.903906 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:36:34.903837    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015afa3f_7b83_4548_974f_f8f0aeb2c7ec.slice/crio-47e633aee476f3dc1da71972542e61c15d98cdf540c3432aeff33fff2ea6bab8 WatchSource:0}: Error finding container 47e633aee476f3dc1da71972542e61c15d98cdf540c3432aeff33fff2ea6bab8: Status 404 returned error can't find the container with id 47e633aee476f3dc1da71972542e61c15d98cdf540c3432aeff33fff2ea6bab8
Oct 30 13:36:34.906199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:34.906171    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"]
Oct 30 13:36:35.274570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:35.274538    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk" event={"ID":"015afa3f-7b83-4548-974f-f8f0aeb2c7ec","Type":"ContainerStarted","Data":"47e633aee476f3dc1da71972542e61c15d98cdf540c3432aeff33fff2ea6bab8"}
Oct 30 13:36:36.283613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:36.283578    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk" event={"ID":"015afa3f-7b83-4548-974f-f8f0aeb2c7ec","Type":"ContainerStarted","Data":"ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110"}
Oct 30 13:36:36.284017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:36.283652    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:36.348639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:36.348590    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk" podStartSLOduration=2.990560659 podStartE2EDuration="3.348575738s" totalImagesPullingTime="358.015079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:36.344455452 +0000 UTC m=+3854.399949274" watchObservedRunningTime="2026-10-30 13:36:36.348575738 +0000 UTC m=+3854.404069561"
Oct 30 13:36:42.297928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:36:42.297899    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:36:42.375653 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:36:42.375610    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:37:12.373716 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:37:12.373674    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:37:38.695486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.695396    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"]
Oct 30 13:37:38.701473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.701449    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:38.719824 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.719792    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"]
Oct 30 13:37:38.726304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.724705    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vs7\" (UniqueName: \"kubernetes.io/projected/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-kube-api-access-x8vs7\") pod \"payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp\" (UID: \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:38.726304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.724765    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp\" (UID: \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:38.730303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.728183    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f97234-dockercfg-hn7sf\""
Oct 30 13:37:38.730303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.728272    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f97234\""
Oct 30 13:37:38.851868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.848609    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vs7\" (UniqueName: \"kubernetes.io/projected/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-kube-api-access-x8vs7\") pod \"payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp\" (UID: \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:38.851868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.848682    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp\" (UID: \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:38.851868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.849763    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp\" (UID: \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:38.886742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.886706    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"]
Oct 30 13:37:38.892102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.892076    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vs7\" (UniqueName: \"kubernetes.io/projected/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-kube-api-access-x8vs7\") pod \"payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp\" (UID: \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:38.899455 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.899424    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"]
Oct 30 13:37:38.899620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.899551    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:38.950702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.950613    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvmq\" (UniqueName: \"kubernetes.io/projected/4e4761cc-99df-4415-9b05-153a79dfc5fc-kube-api-access-qwvmq\") pod \"payload-processing-e2e-inf-b-f97234-69b66665f6-78mks\" (UID: \"4e4761cc-99df-4415-9b05-153a79dfc5fc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:38.951081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:38.951057    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4761cc-99df-4415-9b05-153a79dfc5fc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f97234-69b66665f6-78mks\" (UID: \"4e4761cc-99df-4415-9b05-153a79dfc5fc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:39.038131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.038088    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:39.053546 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.053401    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvmq\" (UniqueName: \"kubernetes.io/projected/4e4761cc-99df-4415-9b05-153a79dfc5fc-kube-api-access-qwvmq\") pod \"payload-processing-e2e-inf-b-f97234-69b66665f6-78mks\" (UID: \"4e4761cc-99df-4415-9b05-153a79dfc5fc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:39.053622 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.053537    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4761cc-99df-4415-9b05-153a79dfc5fc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f97234-69b66665f6-78mks\" (UID: \"4e4761cc-99df-4415-9b05-153a79dfc5fc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:39.054361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.054337    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4761cc-99df-4415-9b05-153a79dfc5fc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f97234-69b66665f6-78mks\" (UID: \"4e4761cc-99df-4415-9b05-153a79dfc5fc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:39.079058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.079020    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvmq\" (UniqueName: \"kubernetes.io/projected/4e4761cc-99df-4415-9b05-153a79dfc5fc-kube-api-access-qwvmq\") pod \"payload-processing-e2e-inf-b-f97234-69b66665f6-78mks\" (UID: \"4e4761cc-99df-4415-9b05-153a79dfc5fc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:39.240611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.240524    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:39.498001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.497546    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"]
Oct 30 13:37:39.562411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.562378    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"]
Oct 30 13:37:39.563138 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:37:39.563112    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4761cc_99df_4415_9b05_153a79dfc5fc.slice/crio-76e3264f3315c9ca6b37c04227a1433adf4e10f1962607ea617c40043d87e0e9 WatchSource:0}: Error finding container 76e3264f3315c9ca6b37c04227a1433adf4e10f1962607ea617c40043d87e0e9: Status 404 returned error can't find the container with id 76e3264f3315c9ca6b37c04227a1433adf4e10f1962607ea617c40043d87e0e9
Oct 30 13:37:39.864580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.864476    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks" event={"ID":"4e4761cc-99df-4415-9b05-153a79dfc5fc","Type":"ContainerStarted","Data":"7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c"}
Oct 30 13:37:39.864580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.864535    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks" event={"ID":"4e4761cc-99df-4415-9b05-153a79dfc5fc","Type":"ContainerStarted","Data":"76e3264f3315c9ca6b37c04227a1433adf4e10f1962607ea617c40043d87e0e9"}
Oct 30 13:37:39.864580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.864564    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:39.866346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.866317    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp" event={"ID":"f3b6e91c-2f71-4503-b75c-4dba639ae8d7","Type":"ContainerStarted","Data":"87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345"}
Oct 30 13:37:39.866480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.866373    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp" event={"ID":"f3b6e91c-2f71-4503-b75c-4dba639ae8d7","Type":"ContainerStarted","Data":"14da98adff52ec7f861095190c06bf7ea163abce1be39e287493e9e89cfa0312"}
Oct 30 13:37:39.866480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.866406    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:39.912980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.912917    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks" podStartSLOduration=1.912902023 podStartE2EDuration="1.912902023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:39.909216223 +0000 UTC m=+3917.964710046" watchObservedRunningTime="2026-10-30 13:37:39.912902023 +0000 UTC m=+3917.968395880"
Oct 30 13:37:39.957213 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:39.957161    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp" podStartSLOduration=1.957147204 podStartE2EDuration="1.957147204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:39.952837189 +0000 UTC m=+3918.008331014" watchObservedRunningTime="2026-10-30 13:37:39.957147204 +0000 UTC m=+3918.012641028"
Oct 30 13:37:42.374604 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:37:42.374566    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:37:42.457032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.457000    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"]
Oct 30 13:37:42.463598 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.463565    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.483910 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.483876    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"]
Oct 30 13:37:42.490540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.490499    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9cc0e6\"/\"openshift-service-ca.crt\""
Oct 30 13:37:42.490540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.490531    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9cc0e6\"/\"kube-root-ca.crt\""
Oct 30 13:37:42.497847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.497819    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9cc0e6\"/\"default-dockercfg-hgnd4\""
Oct 30 13:37:42.497991 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.497879    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9cc0e6\"/\"test-model-9cc0e6-kserve-self-signed-certs\""
Oct 30 13:37:42.525414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.525380    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-model-cache\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.525680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.525643    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-home\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.525778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.525749    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a57ec178-409a-489c-bf08-0da9d6db5da5-tls-certs\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.525858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.525838    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssc4v\" (UniqueName: \"kubernetes.io/projected/a57ec178-409a-489c-bf08-0da9d6db5da5-kube-api-access-ssc4v\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.526033 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.526016    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-dshm\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.526175 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.526159    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-tmp-dir\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.627793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.627703    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-home\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.627793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.627755    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a57ec178-409a-489c-bf08-0da9d6db5da5-tls-certs\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.628049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.627818    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssc4v\" (UniqueName: \"kubernetes.io/projected/a57ec178-409a-489c-bf08-0da9d6db5da5-kube-api-access-ssc4v\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.628049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.627973    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-dshm\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.628158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.628103    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-tmp-dir\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.628214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.628179    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-home\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.628275 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.628213    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-model-cache\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.628522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.628493    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-model-cache\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.628522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.628510    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-tmp-dir\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.630429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.630408    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-dshm\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.630718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.630699    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a57ec178-409a-489c-bf08-0da9d6db5da5-tls-certs\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.656002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.655968    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssc4v\" (UniqueName: \"kubernetes.io/projected/a57ec178-409a-489c-bf08-0da9d6db5da5-kube-api-access-ssc4v\") pod \"test-model-9cc0e6-kserve-596f878b59-8dfqx\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") " pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:42.789908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:42.789862    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:43.110883 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:37:43.110855    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda57ec178_409a_489c_bf08_0da9d6db5da5.slice/crio-eb854d70e01f0040a59f4684c70e66523f5ce1efb35f22399f4fb2e8cd96ca86 WatchSource:0}: Error finding container eb854d70e01f0040a59f4684c70e66523f5ce1efb35f22399f4fb2e8cd96ca86: Status 404 returned error can't find the container with id eb854d70e01f0040a59f4684c70e66523f5ce1efb35f22399f4fb2e8cd96ca86
Oct 30 13:37:43.113129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:43.113112    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"]
Oct 30 13:37:43.908778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:43.908743    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx" event={"ID":"a57ec178-409a-489c-bf08-0da9d6db5da5","Type":"ContainerStarted","Data":"d4e39c78b981d5fd9de496dacb23f57dd4f9f98146755791ff713a451ae02237"}
Oct 30 13:37:43.908778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:43.908787    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx" event={"ID":"a57ec178-409a-489c-bf08-0da9d6db5da5","Type":"ContainerStarted","Data":"eb854d70e01f0040a59f4684c70e66523f5ce1efb35f22399f4fb2e8cd96ca86"}
Oct 30 13:37:43.985109 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:43.985055    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx" podStartSLOduration=1.985038107 podStartE2EDuration="1.985038107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:43.97374319 +0000 UTC m=+3922.029237014" watchObservedRunningTime="2026-10-30 13:37:43.985038107 +0000 UTC m=+3922.040531933"
Oct 30 13:37:44.790686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:44.790652    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:48.703344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.703297    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"]
Oct 30 13:37:48.708860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.708826    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.735751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.735715    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"]
Oct 30 13:37:48.744617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.744582    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9a93bd-openshift-default-dockercfg-mgbm5\""
Oct 30 13:37:48.830923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.830886    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.830923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.830925    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7908707e-27d0-47a1-9633-69f77500fc76-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.831203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.831029    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.831203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.831140    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.831387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.831241    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.831387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.831328    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.831519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.831453    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr4hk\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-kube-api-access-fr4hk\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.831603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.831587    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7908707e-27d0-47a1-9633-69f77500fc76-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.831687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.831671    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.933637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.933590    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.933873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.933714    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.933873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.933796    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.933873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.933826    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.934062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.933922    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr4hk\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-kube-api-access-fr4hk\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.934062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.934043    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7908707e-27d0-47a1-9633-69f77500fc76-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.934181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.934116    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-certs\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.934181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.934132    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.934323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.934249    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.934393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.934319    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7908707e-27d0-47a1-9633-69f77500fc76-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.934730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.934701    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-credential-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.934921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.934894    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-data\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.935083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.935070    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7908707e-27d0-47a1-9633-69f77500fc76-istiod-ca-cert\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.935204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.935188    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-socket\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.936358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.936337    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-envoy\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.937090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.937071    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7908707e-27d0-47a1-9633-69f77500fc76-istio-podinfo\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.961472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.961398    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr4hk\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-kube-api-access-fr4hk\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:48.961760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:48.961736    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-istio-token\") pod \"e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") " pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:49.033156 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:49.033117    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:49.359004 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:37:49.358976    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7908707e_27d0_47a1_9633_69f77500fc76.slice/crio-1e81e05564bdf87015a0bd4935b06a121b48be10a2aa47838396bb14e28807bb WatchSource:0}: Error finding container 1e81e05564bdf87015a0bd4935b06a121b48be10a2aa47838396bb14e28807bb: Status 404 returned error can't find the container with id 1e81e05564bdf87015a0bd4935b06a121b48be10a2aa47838396bb14e28807bb
Oct 30 13:37:49.359777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:49.359749    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"]
Oct 30 13:37:49.362599 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:49.362562    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:49.362696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:49.362650    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:49.362737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:49.362696    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:49.965726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:49.965690    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" event={"ID":"7908707e-27d0-47a1-9633-69f77500fc76","Type":"ContainerStarted","Data":"cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe"}
Oct 30 13:37:49.966156 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:49.965759    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" event={"ID":"7908707e-27d0-47a1-9633-69f77500fc76","Type":"ContainerStarted","Data":"1e81e05564bdf87015a0bd4935b06a121b48be10a2aa47838396bb14e28807bb"}
Oct 30 13:37:50.030401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.030343    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podStartSLOduration=2.030327224 podStartE2EDuration="2.030327224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:50.028222799 +0000 UTC m=+3928.083716623" watchObservedRunningTime="2026-10-30 13:37:50.030327224 +0000 UTC m=+3928.085821048"
Oct 30 13:37:50.033868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.033841    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:50.035956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.035929    4429 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:37:50.036078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.035975    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:37:50.541918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.541885    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"]
Oct 30 13:37:50.547262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.547240    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:50.571305 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.569678    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-af8369\""
Oct 30 13:37:50.571305 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.569843    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-af8369-dockercfg-566tx\""
Oct 30 13:37:50.571604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.571563    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"]
Oct 30 13:37:50.634419 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.634388    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"]
Oct 30 13:37:50.645874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.645845    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:50.743323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.741622    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8\" (UID: \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:50.743323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.741896    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhgf\" (UniqueName: \"kubernetes.io/projected/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-kube-api-access-vbhgf\") pod \"payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8\" (UID: \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:50.861481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.858807    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhgf\" (UniqueName: \"kubernetes.io/projected/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-kube-api-access-vbhgf\") pod \"payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8\" (UID: \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:50.861481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.859029    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8dcaf96-99c3-42c7-8e03-4f45e4433287-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9\" (UID: \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:50.861481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.859085    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8\" (UID: \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:50.861481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.859146    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsshv\" (UniqueName: \"kubernetes.io/projected/b8dcaf96-99c3-42c7-8e03-4f45e4433287-kube-api-access-tsshv\") pod \"payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9\" (UID: \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:50.861481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.861243    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8\" (UID: \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:50.879394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.879344    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"]
Oct 30 13:37:50.894647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.894618    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:37:50.896735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.896708    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:37:50.910773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.910744    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhgf\" (UniqueName: \"kubernetes.io/projected/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-kube-api-access-vbhgf\") pod \"payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8\" (UID: \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:50.976568 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.976531    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8dcaf96-99c3-42c7-8e03-4f45e4433287-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9\" (UID: \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:50.983397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.983365    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsshv\" (UniqueName: \"kubernetes.io/projected/b8dcaf96-99c3-42c7-8e03-4f45e4433287-kube-api-access-tsshv\") pod \"payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9\" (UID: \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:50.983639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:50.978551    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8dcaf96-99c3-42c7-8e03-4f45e4433287-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9\" (UID: \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:51.011597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:51.010813    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsshv\" (UniqueName: \"kubernetes.io/projected/b8dcaf96-99c3-42c7-8e03-4f45e4433287-kube-api-access-tsshv\") pod \"payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9\" (UID: \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:51.033707 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:51.033675    4429 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:37:51.033988 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:51.033735    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:37:51.101321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:51.101250    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:51.184737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:51.184706    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:51.440976 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:37:51.440943    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dcaf96_99c3_42c7_8e03_4f45e4433287.slice/crio-7639e599a882a49bb3189fcbba45b5ce7e0ed0a299f46f17d7738693532a4a6b WatchSource:0}: Error finding container 7639e599a882a49bb3189fcbba45b5ce7e0ed0a299f46f17d7738693532a4a6b: Status 404 returned error can't find the container with id 7639e599a882a49bb3189fcbba45b5ce7e0ed0a299f46f17d7738693532a4a6b
Oct 30 13:37:51.441507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:51.441481    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"]
Oct 30 13:37:51.574587 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:37:51.574556    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0ecdf2_ad30_477b_8374_cdeeb746ec0f.slice/crio-2fccb27ea3fdcf60cbc7955e4003f859af6dcca2be9149ce814fe3bc33dad379 WatchSource:0}: Error finding container 2fccb27ea3fdcf60cbc7955e4003f859af6dcca2be9149ce814fe3bc33dad379: Status 404 returned error can't find the container with id 2fccb27ea3fdcf60cbc7955e4003f859af6dcca2be9149ce814fe3bc33dad379
Oct 30 13:37:51.576140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:51.576116    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"]
Oct 30 13:37:52.004704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.004667    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9" event={"ID":"b8dcaf96-99c3-42c7-8e03-4f45e4433287","Type":"ContainerStarted","Data":"74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b"}
Oct 30 13:37:52.005198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.004723    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9" event={"ID":"b8dcaf96-99c3-42c7-8e03-4f45e4433287","Type":"ContainerStarted","Data":"7639e599a882a49bb3189fcbba45b5ce7e0ed0a299f46f17d7738693532a4a6b"}
Oct 30 13:37:52.005198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.004750    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:37:52.006697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.006670    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8" event={"ID":"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f","Type":"ContainerStarted","Data":"a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c"}
Oct 30 13:37:52.006813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.006719    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8" event={"ID":"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f","Type":"ContainerStarted","Data":"2fccb27ea3fdcf60cbc7955e4003f859af6dcca2be9149ce814fe3bc33dad379"}
Oct 30 13:37:52.006813 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.006764    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:37:52.034123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.034100    4429 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:37:52.034236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.034143    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:37:52.075519 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.075446    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9" podStartSLOduration=2.075426899 podStartE2EDuration="2.075426899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:52.072487398 +0000 UTC m=+3930.127981236" watchObservedRunningTime="2026-10-30 13:37:52.075426899 +0000 UTC m=+3930.130920724"
Oct 30 13:37:52.119828 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.119759    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8" podStartSLOduration=2.119743791 podStartE2EDuration="2.119743791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:52.116177053 +0000 UTC m=+3930.171670881" watchObservedRunningTime="2026-10-30 13:37:52.119743791 +0000 UTC m=+3930.175237616"
Oct 30 13:37:52.790670 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.790631    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:52.803814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:52.803784    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:53.026494 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:53.026469    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:37:53.033909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:53.033885    4429 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:37:53.034004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:53.033937    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:37:54.034215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:54.034184    4429 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:37:54.034651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:54.034239    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:37:55.034451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:55.034418    4429 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:37:55.034899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:55.034474    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:37:56.034405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:56.034321    4429 patch_prober.go:29] interesting pod/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 13:37:56.034405 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:56.034385    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 13:37:57.039858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:57.039826    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:57.040343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:57.040317    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:37:57.041075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:37:57.041055    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:38:03.017639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:03.017607    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:38:03.018194 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:03.018070    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:38:12.373809 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:38:12.373771    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:38:16.768056 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.768026    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"]
Oct 30 13:38:16.772793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.772770    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.800253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.800218    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"]
Oct 30 13:38:16.801819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.801790    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-22efcf\"/\"openshift-service-ca.crt\""
Oct 30 13:38:16.808333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.802476    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-22efcf\"/\"default-dockercfg-hvz9q\""
Oct 30 13:38:16.808497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.802821    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-22efcf\"/\"kube-root-ca.crt\""
Oct 30 13:38:16.813464 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.813440    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-22efcf\"/\"test-model-22efcf-kserve-self-signed-certs\""
Oct 30 13:38:16.819271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.819237    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-home\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.819412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.819377    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-dshm\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.819476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.819446    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-tmp-dir\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.819554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.819533    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad2b06f-919b-40e7-bb40-259fc11e2b97-tls-certs\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.819630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.819587    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-model-cache\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.819681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.819626    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fwtn\" (UniqueName: \"kubernetes.io/projected/dad2b06f-919b-40e7-bb40-259fc11e2b97-kube-api-access-5fwtn\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.920836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.920807    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-dshm\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.921007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.920848    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-tmp-dir\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.921007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.920900    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad2b06f-919b-40e7-bb40-259fc11e2b97-tls-certs\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.921007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.920944    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-model-cache\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.921007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.920973    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fwtn\" (UniqueName: \"kubernetes.io/projected/dad2b06f-919b-40e7-bb40-259fc11e2b97-kube-api-access-5fwtn\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.921262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.921244    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-home\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.921365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.921338    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-tmp-dir\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.921365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.921349    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-model-cache\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.921559 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.921540    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-home\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.923527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.923501    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-dshm\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.923788 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.923769    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad2b06f-919b-40e7-bb40-259fc11e2b97-tls-certs\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:16.945612 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:16.945583    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fwtn\" (UniqueName: \"kubernetes.io/projected/dad2b06f-919b-40e7-bb40-259fc11e2b97-kube-api-access-5fwtn\") pod \"test-model-22efcf-kserve-5bbdcc56f9-stkzj\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") " pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:17.111355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:17.111228    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:17.429465 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:38:17.429424    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad2b06f_919b_40e7_bb40_259fc11e2b97.slice/crio-a87a00cc102825a4e59d522aa76d31f1e572a9b7485f722999be02433ba8753b WatchSource:0}: Error finding container a87a00cc102825a4e59d522aa76d31f1e572a9b7485f722999be02433ba8753b: Status 404 returned error can't find the container with id a87a00cc102825a4e59d522aa76d31f1e572a9b7485f722999be02433ba8753b
Oct 30 13:38:17.441250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:17.441220    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"]
Oct 30 13:38:18.228411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:18.228363    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj" event={"ID":"dad2b06f-919b-40e7-bb40-259fc11e2b97","Type":"ContainerStarted","Data":"90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676"}
Oct 30 13:38:18.228411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:18.228411    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj" event={"ID":"dad2b06f-919b-40e7-bb40-259fc11e2b97","Type":"ContainerStarted","Data":"a87a00cc102825a4e59d522aa76d31f1e572a9b7485f722999be02433ba8753b"}
Oct 30 13:38:18.275214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:18.275163    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj" podStartSLOduration=2.275148619 podStartE2EDuration="2.275148619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:18.273181209 +0000 UTC m=+3956.328675034" watchObservedRunningTime="2026-10-30 13:38:18.275148619 +0000 UTC m=+3956.330642461"
Oct 30 13:38:19.111879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:19.111845    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:27.111961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:27.111930    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:27.125096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:27.125073    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:27.313166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:27.313140    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:38:41.834190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:41.834153    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"]
Oct 30 13:38:41.841147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:41.841120    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:41.858665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:41.858639    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"]
Oct 30 13:38:41.872297 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:41.872249    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7b7407\"/\"test-model-7b7407-kserve-self-signed-certs\""
Oct 30 13:38:41.872476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:41.872350    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7b7407\"/\"default-dockercfg-8qg6w\""
Oct 30 13:38:41.872476 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:41.872402    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7b7407\"/\"openshift-service-ca.crt\""
Oct 30 13:38:41.872786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:41.872761    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7b7407\"/\"kube-root-ca.crt\""
Oct 30 13:38:42.017960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.017927    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tls-certs\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.018160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.018110    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-dshm\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.018327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.018302    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-home\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.018563 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.018539    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-model-cache\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.018685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.018667    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tmp-dir\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.018733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.018717    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2g9\" (UniqueName: \"kubernetes.io/projected/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-kube-api-access-kt2g9\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.120946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.120849    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-dshm\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.121146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.120960    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-home\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.121146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.120992    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-model-cache\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.121146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.121103    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tmp-dir\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.121400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.121169    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2g9\" (UniqueName: \"kubernetes.io/projected/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-kube-api-access-kt2g9\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.121462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.121433    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-home\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.121573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.121469    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-model-cache\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.122432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.122405    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tmp-dir\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.122538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.122526    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tls-certs\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.123647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.123622    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-dshm\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.124954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.124931    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tls-certs\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.147127 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.147104    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2g9\" (UniqueName: \"kubernetes.io/projected/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-kube-api-access-kt2g9\") pod \"test-model-7b7407-kserve-5db7c886cc-c5snp\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") " pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.156037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.156016    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:42.375892 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:38:42.375812    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:38:42.468952 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:38:42.468919    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b02805_60fc_4eee_9f57_52e1d7e1d6b7.slice/crio-4cce6958a43ffb76ce1328ab22032ce8a11c925c97360cc2353ec0cfde96569e WatchSource:0}: Error finding container 4cce6958a43ffb76ce1328ab22032ce8a11c925c97360cc2353ec0cfde96569e: Status 404 returned error can't find the container with id 4cce6958a43ffb76ce1328ab22032ce8a11c925c97360cc2353ec0cfde96569e
Oct 30 13:38:42.471815 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:42.471790    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"]
Oct 30 13:38:43.430883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:43.430844    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp" event={"ID":"21b02805-60fc-4eee-9f57-52e1d7e1d6b7","Type":"ContainerStarted","Data":"db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e"}
Oct 30 13:38:43.431272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:43.430901    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp" event={"ID":"21b02805-60fc-4eee-9f57-52e1d7e1d6b7","Type":"ContainerStarted","Data":"4cce6958a43ffb76ce1328ab22032ce8a11c925c97360cc2353ec0cfde96569e"}
Oct 30 13:38:43.481038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:43.480989    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp" podStartSLOduration=2.4809762060000002 podStartE2EDuration="2.480976206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:43.479310442 +0000 UTC m=+3981.534804264" watchObservedRunningTime="2026-10-30 13:38:43.480976206 +0000 UTC m=+3981.536470029"
Oct 30 13:38:44.156470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:44.156445    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:49.949610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:49.949576    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"]
Oct 30 13:38:49.956131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:49.956111    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:49.973751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:49.973728    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"]
Oct 30 13:38:49.977956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:49.977934    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-af8369-serving-cert\""
Oct 30 13:38:50.038693 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.038656    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.038904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.038882    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.039083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.039063    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8cdf\" (UniqueName: \"kubernetes.io/projected/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-kube-api-access-l8cdf\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.039165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.039150    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-metrics-certs\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.140612 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.140577    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8cdf\" (UniqueName: \"kubernetes.io/projected/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-kube-api-access-l8cdf\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.140766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.140633    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-metrics-certs\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.140766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.140691    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.140882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.140787    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.141423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.141399    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.143180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.143160    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-metrics-certs\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.143353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.143332    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.166007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.165984    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8cdf\" (UniqueName: \"kubernetes.io/projected/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-kube-api-access-l8cdf\") pod \"maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.276619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.276532    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:50.586097 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:38:50.586066    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc20bc63_5f53_491a_9efe_b3f5f02e59dd.slice/crio-cd77cd04a058e7e46c67d5639bb7057e33389539f3f326da115147257c2a70c5 WatchSource:0}: Error finding container cd77cd04a058e7e46c67d5639bb7057e33389539f3f326da115147257c2a70c5: Status 404 returned error can't find the container with id cd77cd04a058e7e46c67d5639bb7057e33389539f3f326da115147257c2a70c5
Oct 30 13:38:50.587512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.587492    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:38:50.588030 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:50.588001    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"]
Oct 30 13:38:51.496887 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:51.496839    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5" event={"ID":"fc20bc63-5f53-491a-9efe-b3f5f02e59dd","Type":"ContainerStarted","Data":"a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9"}
Oct 30 13:38:51.497330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:51.496907    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5" event={"ID":"fc20bc63-5f53-491a-9efe-b3f5f02e59dd","Type":"ContainerStarted","Data":"cd77cd04a058e7e46c67d5639bb7057e33389539f3f326da115147257c2a70c5"}
Oct 30 13:38:51.497330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:51.496962    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:38:51.552206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:51.552140    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5" podStartSLOduration=2.211529166 podStartE2EDuration="2.552126533s" totalImagesPullingTime="340.597367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:51.548832075 +0000 UTC m=+3989.604325900" watchObservedRunningTime="2026-10-30 13:38:51.552126533 +0000 UTC m=+3989.607620358"
Oct 30 13:38:52.156376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:52.156324    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:52.168978 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:52.168954    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:52.518023 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:52.517989    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:38:57.512125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:38:57.512096    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:39:12.373087 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:39:12.373004    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:39:42.372810 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:39:42.372778    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:39:52.774863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.774824    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"]
Oct 30 13:39:52.780204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.780184    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.803320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.803271    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"]
Oct 30 13:39:52.810448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.810406    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7b7407-openshift-default-dockercfg-99s4b\""
Oct 30 13:39:52.947371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.947317    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-envoy\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.947588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.947439    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-certs\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.947588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.947494    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.947699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.947627    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-token\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.947880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.947859    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-podinfo\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.947953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.947931    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-credential-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.948008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.947984    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-data\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.948097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.948075    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvg6\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-kube-api-access-ncvg6\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:52.948139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:52.948115    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8475e8-fba5-4fda-9538-1ae39efe931e-istiod-ca-cert\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049177    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-podinfo\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049244    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-credential-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049267    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-data\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049338    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvg6\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-kube-api-access-ncvg6\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049374    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8475e8-fba5-4fda-9538-1ae39efe931e-istiod-ca-cert\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049430    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-envoy\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049483    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-certs\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049553 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049522    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049910 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049589    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-token\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.049973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.049907    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-data\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.050648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.050365    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8475e8-fba5-4fda-9538-1ae39efe931e-istiod-ca-cert\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.050648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.050556    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-credential-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.050648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.050611    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-certs\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.050851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.050713    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-socket\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.052268 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.052245    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-podinfo\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.052408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.052330    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-envoy\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.075054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.075030    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-token\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.075554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.075533    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvg6\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-kube-api-access-ncvg6\") pod \"e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") " pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.096042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.095867    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:53.428416 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:39:53.428386    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8475e8_fba5_4fda_9538_1ae39efe931e.slice/crio-1bd83e6f5c63fc49bfd32b407671d8f5c871f41ba320a0d5eca1adb6249df53a WatchSource:0}: Error finding container 1bd83e6f5c63fc49bfd32b407671d8f5c871f41ba320a0d5eca1adb6249df53a: Status 404 returned error can't find the container with id 1bd83e6f5c63fc49bfd32b407671d8f5c871f41ba320a0d5eca1adb6249df53a
Oct 30 13:39:53.431461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.431436    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"]
Oct 30 13:39:53.432094 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.432062    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:39:53.432146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.432135    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:39:53.432182 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:53.432166    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:39:54.089941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:54.089902    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" event={"ID":"5a8475e8-fba5-4fda-9538-1ae39efe931e","Type":"ContainerStarted","Data":"1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb"}
Oct 30 13:39:54.090440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:54.089973    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" event={"ID":"5a8475e8-fba5-4fda-9538-1ae39efe931e","Type":"ContainerStarted","Data":"1bd83e6f5c63fc49bfd32b407671d8f5c871f41ba320a0d5eca1adb6249df53a"}
Oct 30 13:39:54.096463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:54.096432    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:39:54.098460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:54.098430    4429 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:39:54.098586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:54.098486    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:39:54.137781 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:54.137720    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podStartSLOduration=2.137703306 podStartE2EDuration="2.137703306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:54.1340766 +0000 UTC m=+4052.189570427" watchObservedRunningTime="2026-10-30 13:39:54.137703306 +0000 UTC m=+4052.193197132"
Oct 30 13:39:55.096498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:55.096462    4429 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:39:55.096905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:55.096518    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:39:56.096413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:56.096329    4429 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:39:56.096413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:56.096394    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:39:57.096324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:57.096249    4429 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:39:57.096541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:57.096346    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:39:58.096572 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:58.096537    4429 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:39:58.097112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:58.096597    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:39:59.096868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:59.096833    4429 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 13:39:59.097336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:39:59.096890    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 13:40:00.186425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:00.186382    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:40:00.187626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:00.187601    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:40:00.188147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:00.188122    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:40:01.190900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.190862    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"]
Oct 30 13:40:01.196844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.196816    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.219512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.219482    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"]
Oct 30 13:40:01.300454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.300420    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f97234-openshift-default-dockercfg-x8qzl\""
Oct 30 13:40:01.328215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328178    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.328454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328401    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.328525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328477    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pwnf\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-kube-api-access-2pwnf\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.328628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328605    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.328672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328652    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.328759 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328744    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.328870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328843    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.329018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328893    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.329018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.328940    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430014    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430050    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430110    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430169    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430189    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430220    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430254    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430433    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430516    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pwnf\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-kube-api-access-2pwnf\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430508    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430530    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430862    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.430900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.430896    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.431166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.431140    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.432751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.432729    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.433341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.433321    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.456182 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.456100    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.456182 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.456123    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pwnf\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-kube-api-access-2pwnf\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.529692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.529640    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:01.866939 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:40:01.866900    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b4e94b_6359_4b53_8bbb_501c9cd698ce.slice/crio-8d2a00e9fe8b36e7e90d9a7b2306bbfe864327947263266bbf4d9a52d9ec861f WatchSource:0}: Error finding container 8d2a00e9fe8b36e7e90d9a7b2306bbfe864327947263266bbf4d9a52d9ec861f: Status 404 returned error can't find the container with id 8d2a00e9fe8b36e7e90d9a7b2306bbfe864327947263266bbf4d9a52d9ec861f
Oct 30 13:40:01.868580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.868553    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"]
Oct 30 13:40:01.870450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.870416    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:40:01.870534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.870489    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:40:01.870534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:01.870519    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:40:02.159854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:02.159768    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" event={"ID":"31b4e94b-6359-4b53-8bbb-501c9cd698ce","Type":"ContainerStarted","Data":"9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51"}
Oct 30 13:40:02.159854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:02.159820    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" event={"ID":"31b4e94b-6359-4b53-8bbb-501c9cd698ce","Type":"ContainerStarted","Data":"8d2a00e9fe8b36e7e90d9a7b2306bbfe864327947263266bbf4d9a52d9ec861f"}
Oct 30 13:40:02.211103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:02.211041    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podStartSLOduration=1.211023353 podStartE2EDuration="1.211023353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:02.209207316 +0000 UTC m=+4060.264701140" watchObservedRunningTime="2026-10-30 13:40:02.211023353 +0000 UTC m=+4060.266517176"
Oct 30 13:40:02.529946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:02.529907    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:02.531639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:02.531613    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 13:40:02.531779 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:02.531668    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 13:40:03.534218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:03.530124    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 13:40:03.534218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:03.530181    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 13:40:04.530666 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:04.530628    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 13:40:04.530855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:04.530695    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 13:40:05.530126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:05.530096    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 13:40:05.530561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:05.530157    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 13:40:06.530036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:06.530000    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 13:40:06.530189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:06.530061    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 13:40:07.530393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:07.530350    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 13:40:07.530866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:07.530423    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 13:40:08.555163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:08.555117    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:08.555945 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:08.555911    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:08.557690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:08.557654    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:40:12.374177 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:40:12.374141    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:40:31.300730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.300689    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"]
Oct 30 13:40:31.306230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.306206    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.360394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.360356    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"]
Oct 30 13:40:31.481570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.481511    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.481819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.481665    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.481819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.481696    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.481819 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.481810    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.482004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.481889    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.482004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.481934    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.482099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.482073    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.482155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.482116    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.482155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.482133    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxr7r\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-kube-api-access-sxr7r\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.583659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.583573    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.583860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.583669    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.583860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.583704    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.583860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.583767    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.583860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.583834    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.583877    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.583976    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.584029    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.584037    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-certs\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584090 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.584057    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxr7r\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-kube-api-access-sxr7r\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.584349    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-credential-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.584613    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-socket\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.584728    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istiod-ca-cert\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.584879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.584859    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-data\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.586613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.586583    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-podinfo\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.586818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.586800    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-envoy\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.613346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.613322    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-token\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.613516 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.613498    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxr7r\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-kube-api-access-sxr7r\") pod \"e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") " pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.626628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.626604    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:31.961203 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:40:31.961171    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84945ac_ece7_47e8_8bd7_3eb6bbba1c32.slice/crio-3798c3d4d972543dbecea2fe032e33c258af1fbd681bc391f8dc47c719797563 WatchSource:0}: Error finding container 3798c3d4d972543dbecea2fe032e33c258af1fbd681bc391f8dc47c719797563: Status 404 returned error can't find the container with id 3798c3d4d972543dbecea2fe032e33c258af1fbd681bc391f8dc47c719797563
Oct 30 13:40:31.963316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.963271    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"]
Oct 30 13:40:31.965045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.965009    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:40:31.965130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.965083    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:40:31.965130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:31.965113    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:40:32.430835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:32.430736    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" event={"ID":"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32","Type":"ContainerStarted","Data":"3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e"}
Oct 30 13:40:32.430835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:32.430814    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" event={"ID":"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32","Type":"ContainerStarted","Data":"3798c3d4d972543dbecea2fe032e33c258af1fbd681bc391f8dc47c719797563"}
Oct 30 13:40:32.490885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:32.490821    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" podStartSLOduration=1.490798925 podStartE2EDuration="1.490798925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:32.485008288 +0000 UTC m=+4090.540502137" watchObservedRunningTime="2026-10-30 13:40:32.490798925 +0000 UTC m=+4090.546292750"
Oct 30 13:40:32.627392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:32.627350    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:33.629219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:33.629141    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 13:40:33.629219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:33.629195    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 13:40:34.627543 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:34.627508    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 13:40:34.627728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:34.627565    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 13:40:35.627642 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:35.627606    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 13:40:35.628069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:35.627675    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 13:40:36.627435 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:36.627403    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 13:40:36.627623 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:36.627459    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 13:40:37.627991 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:37.627948    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 13:40:37.628437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:37.628016    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 13:40:38.633253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:38.633212    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:38.633758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:38.633736    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:38.634521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:40:38.634499    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:40:42.375534 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:40:42.375493    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:41:02.135485 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:02.135447    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"]
Oct 30 13:41:02.136001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:02.135774    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy" containerID="cri-o://e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498" gracePeriod=30
Oct 30 13:41:04.462793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:04.462761    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"]
Oct 30 13:41:04.463335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:04.462973    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy" containerID="cri-o://532d887086c7700d6ec37fe02f59caa329da2e67d9c208bdf2aa78de324178b4" gracePeriod=30
Oct 30 13:41:07.453998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.453964    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:41:07.527653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.527625    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-kube-api-access-kgcgl\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-kube-api-access-kgcgl\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.527856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.527727    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-credential-socket\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.527856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.527829    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-data\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.527998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.527888    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-envoy\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.527998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.527937    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a128d702-ca8a-455f-8ec0-42244bba082b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a128d702-ca8a-455f-8ec0-42244bba082b-istiod-ca-cert\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.527998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.527984    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a128d702-ca8a-455f-8ec0-42244bba082b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a128d702-ca8a-455f-8ec0-42244bba082b-istio-podinfo\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.528196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.528022    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-socket\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.528196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.528113    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-credential-socket" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:07.528314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.528198    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-istio-token\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-istio-token\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.528314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.528215    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-data" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:07.528482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.528408    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-certs\") pod \"a128d702-ca8a-455f-8ec0-42244bba082b\" (UID: \"a128d702-ca8a-455f-8ec0-42244bba082b\") "
Oct 30 13:41:07.528482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.528433    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-socket" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:07.528705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.528679    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a128d702-ca8a-455f-8ec0-42244bba082b-istiod-ca-cert" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:07.528705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.528698    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-certs" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:07.529270 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.529247    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.529378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.529275    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.529378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.529306    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.529378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.529319    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a128d702-ca8a-455f-8ec0-42244bba082b-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.529378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.529333    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.531203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.531155    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-istio-token" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:07.531363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.531340    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-envoy" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:07.531439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.531354    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a128d702-ca8a-455f-8ec0-42244bba082b-istio-podinfo" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:07.531851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.531825    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-kube-api-access-kgcgl" pod "a128d702-ca8a-455f-8ec0-42244bba082b" (UID: "a128d702-ca8a-455f-8ec0-42244bba082b"). InnerVolumeSpecName "kube-api-access-kgcgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:07.631355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.631319    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.631355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.631352    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcgl\" (UniqueName: \"kubernetes.io/projected/a128d702-ca8a-455f-8ec0-42244bba082b-kube-api-access-kgcgl\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.631355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.631363    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a128d702-ca8a-455f-8ec0-42244bba082b-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.631620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.631371    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a128d702-ca8a-455f-8ec0-42244bba082b-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:07.792430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.792395    4429 generic.go:356] "Generic (PLEG): container finished" podID="a128d702-ca8a-455f-8ec0-42244bba082b" containerID="e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498" exitCode=0
Oct 30 13:41:07.792633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.792446    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"
Oct 30 13:41:07.792633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.792494    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" event={"ID":"a128d702-ca8a-455f-8ec0-42244bba082b","Type":"ContainerDied","Data":"e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498"}
Oct 30 13:41:07.792633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.792557    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4" event={"ID":"a128d702-ca8a-455f-8ec0-42244bba082b","Type":"ContainerDied","Data":"65789f3e847d254a89ccb6839f74c5f632ba466b4e59060d13b160917c2a476a"}
Oct 30 13:41:07.792633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.792585    4429 scope.go:122] "RemoveContainer" containerID="e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498"
Oct 30 13:41:07.807351 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.807331    4429 scope.go:122] "RemoveContainer" containerID="e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498"
Oct 30 13:41:07.807712 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:41:07.807691    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498\": container with ID starting with e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498 not found: ID does not exist" containerID="e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498"
Oct 30 13:41:07.807832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.807721    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498"} err="failed to get container status \"e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498\": rpc error: code = NotFound desc = could not find container \"e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498\": container with ID starting with e4816752eefa919ecdb3df3da3261ed60c8c3ef9e3704cb35beb30aef4e53498 not found: ID does not exist"
Oct 30 13:41:07.853741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.853705    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"]
Oct 30 13:41:07.866413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:07.866373    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-63499e-openshift-default-6df6b584fc-xbqc4"]
Oct 30 13:41:08.078927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:08.078847    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" path="/var/lib/kubelet/pods/a128d702-ca8a-455f-8ec0-42244bba082b/volumes"
Oct 30 13:41:08.629731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:08.629688    4429 patch_prober.go:29] interesting pod/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:41:08.630229 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:08.629766    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:41:09.471129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:09.471093    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"]
Oct 30 13:41:09.471381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:09.471348    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy" containerID="cri-o://b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882" gracePeriod=30
Oct 30 13:41:09.846017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:09.845924    4429 generic.go:356] "Generic (PLEG): container finished" podID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerID="532d887086c7700d6ec37fe02f59caa329da2e67d9c208bdf2aa78de324178b4" exitCode=0
Oct 30 13:41:09.848491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:09.848457    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" event={"ID":"bc56bee0-eb57-46f5-b67a-2fa923d073c2","Type":"ContainerDied","Data":"532d887086c7700d6ec37fe02f59caa329da2e67d9c208bdf2aa78de324178b4"}
Oct 30 13:41:09.902847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:09.902819    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:41:10.011730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.011688    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istiod-ca-cert\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.012527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.012500    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-certs\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.013109 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.012406    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istiod-ca-cert" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:10.013639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.012960    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-certs" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:10.013822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.013784    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-podinfo\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.014018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.014001    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-kube-api-access-jkdsg\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-kube-api-access-jkdsg\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.014172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.014153    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-envoy\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.014689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.014672    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-data\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.034355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.015222    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-data" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:10.034833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.017374    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-podinfo" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:10.034980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.034592    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-socket\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.035639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.035046    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-socket" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:10.035965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.035944    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-credential-socket\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.036145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.036129    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-token\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-token\") pod \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\" (UID: \"bc56bee0-eb57-46f5-b67a-2fa923d073c2\") "
Oct 30 13:41:10.037928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.037805    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.038398 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.038383    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.038645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.038632    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.038797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.038785    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.038916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.038904    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.039013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.038342    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-credential-socket" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:10.048017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.047964    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-token" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:10.055227 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.054486    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-kube-api-access-jkdsg" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "kube-api-access-jkdsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:10.058040 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.057961    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-envoy" pod "bc56bee0-eb57-46f5-b67a-2fa923d073c2" (UID: "bc56bee0-eb57-46f5-b67a-2fa923d073c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:10.141323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.141234    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdsg\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-kube-api-access-jkdsg\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.141323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.141324    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.141618 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.141339    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc56bee0-eb57-46f5-b67a-2fa923d073c2-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.141618 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.141351    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc56bee0-eb57-46f5-b67a-2fa923d073c2-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:10.858463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.858382    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"
Oct 30 13:41:10.858915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.858381    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757" event={"ID":"bc56bee0-eb57-46f5-b67a-2fa923d073c2","Type":"ContainerDied","Data":"66a050042bedc6fe2d1ebfa9a0dfb23abaa1fd20b7278f68099dfe0eb0f8e124"}
Oct 30 13:41:10.858915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.858515    4429 scope.go:122] "RemoveContainer" containerID="532d887086c7700d6ec37fe02f59caa329da2e67d9c208bdf2aa78de324178b4"
Oct 30 13:41:10.933357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.933325    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"]
Oct 30 13:41:10.946845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:10.946815    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-75d385-openshift-default-8785c777c-zt757"]
Oct 30 13:41:12.079230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:12.079191    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" path="/var/lib/kubelet/pods/bc56bee0-eb57-46f5-b67a-2fa923d073c2/volumes"
Oct 30 13:41:12.375012 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:41:12.374915    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:41:14.788494 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.788462    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:41:14.835035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835004    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-podinfo\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835076    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-credential-socket\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835212    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-socket\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835346    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-kube-api-access-kz8rk\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-kube-api-access-kz8rk\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835390    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istiod-ca-cert\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835439    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-credential-socket" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:14.835562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835481    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-socket" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:14.835562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835542    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-envoy\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835570    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-certs\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835620    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-data\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835767    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-token\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-token\") pod \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\" (UID: \"d4d53a6d-0f35-4bb9-b1ec-77df3434474a\") "
Oct 30 13:41:14.835961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835933    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istiod-ca-cert" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:14.836074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.835999    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-certs" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:14.836414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.836362    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-data" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:14.836914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.836879    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.836914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.836915    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.837091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.836929    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.837091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.836942    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.837091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.836955    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.837969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.837920    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-podinfo" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:14.838112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.837933    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-kube-api-access-kz8rk" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "kube-api-access-kz8rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:14.838431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.838411    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-envoy" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:14.838589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.838573    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-token" pod "d4d53a6d-0f35-4bb9-b1ec-77df3434474a" (UID: "d4d53a6d-0f35-4bb9-b1ec-77df3434474a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:14.901687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.901639    4429 generic.go:356] "Generic (PLEG): container finished" podID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerID="b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882" exitCode=0
Oct 30 13:41:14.901899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.901717    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" event={"ID":"d4d53a6d-0f35-4bb9-b1ec-77df3434474a","Type":"ContainerDied","Data":"b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882"}
Oct 30 13:41:14.901899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.901751    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"
Oct 30 13:41:14.901899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.901793    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w" event={"ID":"d4d53a6d-0f35-4bb9-b1ec-77df3434474a","Type":"ContainerDied","Data":"fb5180a5fb605da2a1b1d98b04e8b87acce30bcf5c970f9cd1506206af7f9685"}
Oct 30 13:41:14.901899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.901820    4429 scope.go:122] "RemoveContainer" containerID="b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882"
Oct 30 13:41:14.917938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.917908    4429 scope.go:122] "RemoveContainer" containerID="b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882"
Oct 30 13:41:14.918360 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:41:14.918270    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882\": container with ID starting with b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882 not found: ID does not exist" containerID="b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882"
Oct 30 13:41:14.918360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.918327    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882"} err="failed to get container status \"b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882\": rpc error: code = NotFound desc = could not find container \"b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882\": container with ID starting with b503cfd9be9b415b198b9b3acb812323e9b967459d076c431166a8eba47d6882 not found: ID does not exist"
Oct 30 13:41:14.938334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.938302    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz8rk\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-kube-api-access-kz8rk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.938334 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.938334    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.938500 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.938348    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.938500 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.938356    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d53a6d-0f35-4bb9-b1ec-77df3434474a-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:14.972853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.972818    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"]
Oct 30 13:41:14.978123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:14.978091    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d9115-openshift-default-856bf8bb69-kfk8w"]
Oct 30 13:41:15.117763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:15.117668    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"]
Oct 30 13:41:15.117963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:15.117908    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy" containerID="cri-o://b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3" gracePeriod=30
Oct 30 13:41:16.078558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:16.078517    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" path="/var/lib/kubelet/pods/d4d53a6d-0f35-4bb9-b1ec-77df3434474a/volumes"
Oct 30 13:41:20.421081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.421041    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:41:20.529888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.529846    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-data\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.529945    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-envoy\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530023    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/582c6289-e704-4dbb-869d-42e329231c15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582c6289-e704-4dbb-869d-42e329231c15-istio-podinfo\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530150    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-data" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.530256 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530221    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-socket\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530340    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-certs\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530427    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582c6289-e704-4dbb-869d-42e329231c15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582c6289-e704-4dbb-869d-42e329231c15-istiod-ca-cert\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530472    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-socket" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.530569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530486    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-kube-api-access-q7rcq\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-kube-api-access-q7rcq\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530527    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-istio-token\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-istio-token\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530624    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-certs" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.530720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530673    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-credential-socket\") pod \"582c6289-e704-4dbb-869d-42e329231c15\" (UID: \"582c6289-e704-4dbb-869d-42e329231c15\") "
Oct 30 13:41:20.530838 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530783    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582c6289-e704-4dbb-869d-42e329231c15-istiod-ca-cert" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:20.530909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.530890    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-credential-socket" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.531511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.531484    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.531511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.531512    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.531718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.531524    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.531718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.531536    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582c6289-e704-4dbb-869d-42e329231c15-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.531718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.531549    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.532657 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.532632    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/582c6289-e704-4dbb-869d-42e329231c15-istio-podinfo" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:20.533198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.533179    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-istio-token" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:20.533361 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.533340    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-envoy" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:20.533427 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.533407    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-kube-api-access-q7rcq" pod "582c6289-e704-4dbb-869d-42e329231c15" (UID: "582c6289-e704-4dbb-869d-42e329231c15"). InnerVolumeSpecName "kube-api-access-q7rcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:20.632906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.632871    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582c6289-e704-4dbb-869d-42e329231c15-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.632906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.632903    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582c6289-e704-4dbb-869d-42e329231c15-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.632906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.632912    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7rcq\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-kube-api-access-q7rcq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.633242 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.632923    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582c6289-e704-4dbb-869d-42e329231c15-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:20.976638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.976600    4429 generic.go:356] "Generic (PLEG): container finished" podID="582c6289-e704-4dbb-869d-42e329231c15" containerID="b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3" exitCode=0
Oct 30 13:41:20.976869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.976671    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"
Oct 30 13:41:20.976869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.976699    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" event={"ID":"582c6289-e704-4dbb-869d-42e329231c15","Type":"ContainerDied","Data":"b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3"}
Oct 30 13:41:20.976869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.976751    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx" event={"ID":"582c6289-e704-4dbb-869d-42e329231c15","Type":"ContainerDied","Data":"5312314b972232fd1594ff31d935c0a207d28a8ad14818b66bba5051ec9ad123"}
Oct 30 13:41:20.976869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.976780    4429 scope.go:122] "RemoveContainer" containerID="b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3"
Oct 30 13:41:20.997870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.997813    4429 scope.go:122] "RemoveContainer" containerID="b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3"
Oct 30 13:41:20.998334 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:41:20.998300    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3\": container with ID starting with b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3 not found: ID does not exist" containerID="b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3"
Oct 30 13:41:20.998474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:20.998342    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3"} err="failed to get container status \"b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3\": rpc error: code = NotFound desc = could not find container \"b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3\": container with ID starting with b78afe09bb9e8a88d6b0f8c6d9576c3245b80d3e46c4f32f6836e4c5567639f3 not found: ID does not exist"
Oct 30 13:41:21.041664 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.041628    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"]
Oct 30 13:41:21.053260 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.053221    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-jhgnx"]
Oct 30 13:41:21.767772 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.767736    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"]
Oct 30 13:41:21.768917 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.768896    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy"
Oct 30 13:41:21.768917 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.768917    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy"
Oct 30 13:41:21.769104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.768947    4429 cpu_manager.go:414] "containerMap: removing container" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy"
Oct 30 13:41:21.769104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.768954    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy"
Oct 30 13:41:21.769104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.768962    4429 cpu_manager.go:414] "containerMap: removing container" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy"
Oct 30 13:41:21.769104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.768967    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy"
Oct 30 13:41:21.769104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.768973    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy"
Oct 30 13:41:21.769104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.768978    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy"
Oct 30 13:41:21.769448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.769131    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a128d702-ca8a-455f-8ec0-42244bba082b" containerName="istio-proxy"
Oct 30 13:41:21.769448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.769142    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc56bee0-eb57-46f5-b67a-2fa923d073c2" containerName="istio-proxy"
Oct 30 13:41:21.769448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.769148    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="582c6289-e704-4dbb-869d-42e329231c15" containerName="istio-proxy"
Oct 30 13:41:21.769448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.769156    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d53a6d-0f35-4bb9-b1ec-77df3434474a" containerName="istio-proxy"
Oct 30 13:41:21.775068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.775034    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.792960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.792927    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-09ad1e-openshift-default-dockercfg-h2w8d\""
Oct 30 13:41:21.813376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.813340    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"]
Oct 30 13:41:21.852736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.852694    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.853024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.852786    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-data\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.853024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.852895    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-token\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.853181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.853121    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87bx\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-kube-api-access-d87bx\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.853181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.853166    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-podinfo\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.853331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.853309    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-credential-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.853502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.853476    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-envoy\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.853613 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.853596    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-certs\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.853723 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.853708    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istiod-ca-cert\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.954826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.954790    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.954855    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-data\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.954887    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-token\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.955012    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87bx\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-kube-api-access-d87bx\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.955037    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-podinfo\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.955079    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-credential-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.955131    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-envoy\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.955187    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-certs\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.955242    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istiod-ca-cert\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.955919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.955896    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istiod-ca-cert\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.956118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.956103    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.956298 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.956268    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-data\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.956846 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.956817    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-credential-socket\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.959076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.959046    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-envoy\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.960252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.959477    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-podinfo\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.960252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.959831    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-certs\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.983266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.983201    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87bx\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-kube-api-access-d87bx\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:21.983266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:21.983231    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-token\") pod \"e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") " pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:22.080650 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:22.080570    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582c6289-e704-4dbb-869d-42e329231c15" path="/var/lib/kubelet/pods/582c6289-e704-4dbb-869d-42e329231c15/volumes"
Oct 30 13:41:22.092545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:22.092510    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:22.445549 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:41:22.445503    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce57651f_cec8_4d1b_b46d_04b081bd0ac8.slice/crio-32d22b4e8332711b06c3a76ceb63215c9dea4f55b265e04d3a6bf641d05e5e96 WatchSource:0}: Error finding container 32d22b4e8332711b06c3a76ceb63215c9dea4f55b265e04d3a6bf641d05e5e96: Status 404 returned error can't find the container with id 32d22b4e8332711b06c3a76ceb63215c9dea4f55b265e04d3a6bf641d05e5e96
Oct 30 13:41:22.446976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:22.446951    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"]
Oct 30 13:41:22.449410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:22.449371    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:41:22.449490 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:22.449450    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:41:22.449490 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:22.449481    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:41:23.006359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:23.006323    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" event={"ID":"ce57651f-cec8-4d1b-b46d-04b081bd0ac8","Type":"ContainerStarted","Data":"7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523"}
Oct 30 13:41:23.006764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:23.006401    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" event={"ID":"ce57651f-cec8-4d1b-b46d-04b081bd0ac8","Type":"ContainerStarted","Data":"32d22b4e8332711b06c3a76ceb63215c9dea4f55b265e04d3a6bf641d05e5e96"}
Oct 30 13:41:23.060965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:23.060905    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podStartSLOduration=2.060888767 podStartE2EDuration="2.060888767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:23.059930165 +0000 UTC m=+4141.115424011" watchObservedRunningTime="2026-10-30 13:41:23.060888767 +0000 UTC m=+4141.116382591"
Oct 30 13:41:23.093497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:23.093462    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:23.095247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:23.095221    4429 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 13:41:23.095424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:23.095274    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 13:41:24.093924 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:24.093889    4429 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 13:41:24.094374 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:24.093945    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 13:41:25.093017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:25.092980    4429 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 13:41:25.093219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:25.093040    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 13:41:26.093112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:26.093079    4429 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 13:41:26.093580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:26.093136    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 13:41:27.093891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:27.093857    4429 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 13:41:27.094358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:27.093915    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 13:41:28.093575 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:28.093534    4429 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 13:41:28.093805 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:28.093598    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 13:41:29.093946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:29.093909    4429 patch_prober.go:29] interesting pod/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 13:41:29.094372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:29.093968    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 13:41:30.099455 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:30.099426    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:30.099956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:30.099703    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:30.100497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:30.100478    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:41:31.633081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:31.633047    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"]
Oct 30 13:41:31.674184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:31.674145    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"]
Oct 30 13:41:36.633748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:36.633700    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg" podUID="9ad53602-bf0d-4ca4-9261-e515e8b05e4e" containerName="payload-processing" containerID="cri-o://29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3" gracePeriod=25
Oct 30 13:41:36.675007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:36.674960    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d" podUID="4f35f65b-2202-4bd9-b8ea-705c55d37960" containerName="payload-pre-processing" containerID="cri-o://c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1" gracePeriod=25
Oct 30 13:41:39.985333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:39.985299    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"]
Oct 30 13:41:39.990929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:39.990906    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.012464 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.012427    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"]
Oct 30 13:41:40.028929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.020982    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-22efcf-openshift-default-dockercfg-rh6lg\""
Oct 30 13:41:40.031400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.031369    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg" podUID="9ad53602-bf0d-4ca4-9261-e515e8b05e4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:41:40.031400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.031385    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d" podUID="4f35f65b-2202-4bd9-b8ea-705c55d37960" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:41:40.099027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.098991    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-data\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.099223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.099117    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-envoy\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.099223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.099175    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8nd8\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-kube-api-access-f8nd8\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.099430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.099360    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.099430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.099398    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920d1596-271c-4f07-bd46-061f9adc053d-istiod-ca-cert\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.099566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.099545    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-istio-token\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.099680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.099667    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920d1596-271c-4f07-bd46-061f9adc053d-istio-podinfo\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.099758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.099742    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-certs\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.099838 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.099824    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-credential-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201116    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8nd8\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-kube-api-access-f8nd8\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201213    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201236    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920d1596-271c-4f07-bd46-061f9adc053d-istiod-ca-cert\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201331    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-istio-token\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201427    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920d1596-271c-4f07-bd46-061f9adc053d-istio-podinfo\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201507    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-certs\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201559    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-credential-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201662    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201681    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-data\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201788    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-envoy\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.201969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201834    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-certs\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.202012 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.201965    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-credential-socket\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.202051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.202036    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-data\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.202189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.202166    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920d1596-271c-4f07-bd46-061f9adc053d-istiod-ca-cert\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.204186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.204161    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-envoy\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.204327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.204235    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920d1596-271c-4f07-bd46-061f9adc053d-istio-podinfo\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.226337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.226302    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-istio-token\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.226619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.226594    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8nd8\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-kube-api-access-f8nd8\") pod \"e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") " pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.308566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.308471    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:40.638384 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:41:40.638354    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920d1596_271c_4f07_bd46_061f9adc053d.slice/crio-9ac26280e13c69a1d102c9739939a3d76ba1722325381630b79466abff4c93b9 WatchSource:0}: Error finding container 9ac26280e13c69a1d102c9739939a3d76ba1722325381630b79466abff4c93b9: Status 404 returned error can't find the container with id 9ac26280e13c69a1d102c9739939a3d76ba1722325381630b79466abff4c93b9
Oct 30 13:41:40.639688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.639666    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"]
Oct 30 13:41:40.641690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.641654    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:41:40.641812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.641739    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:41:40.641812 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:40.641785    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:41:41.188195 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:41.188155    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" event={"ID":"920d1596-271c-4f07-bd46-061f9adc053d","Type":"ContainerStarted","Data":"fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77"}
Oct 30 13:41:41.188671 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:41.188214    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" event={"ID":"920d1596-271c-4f07-bd46-061f9adc053d","Type":"ContainerStarted","Data":"9ac26280e13c69a1d102c9739939a3d76ba1722325381630b79466abff4c93b9"}
Oct 30 13:41:41.259307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:41.258193    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" podStartSLOduration=2.258154531 podStartE2EDuration="2.258154531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:41.253515709 +0000 UTC m=+4159.309009555" watchObservedRunningTime="2026-10-30 13:41:41.258154531 +0000 UTC m=+4159.313648357"
Oct 30 13:41:41.308784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:41.308749    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:41.311162 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:41.311136    4429 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 13:41:41.311307 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:41.311187    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 13:41:42.309523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:42.309488    4429 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 13:41:42.309961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:42.309550    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 13:41:42.397868 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:41:42.397821    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:41:43.309820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:43.309358    4429 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 13:41:43.309820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:43.309420    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 13:41:44.309473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:44.309437    4429 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 13:41:44.309681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:44.309496    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 13:41:45.309803 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:45.309767    4429 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 13:41:45.310354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:45.309828    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 13:41:46.309122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:46.309075    4429 patch_prober.go:29] interesting pod/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 30 13:41:46.309423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:46.309155    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 30 13:41:47.315791 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:47.315754    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:47.316344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:47.316248    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:47.317064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:47.317043    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:41:50.032341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:50.032266    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg" podUID="9ad53602-bf0d-4ca4-9261-e515e8b05e4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:41:50.032807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:50.032570    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d" podUID="4f35f65b-2202-4bd9-b8ea-705c55d37960" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:41:56.968927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:56.968891    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:41:57.015207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.015173    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-plugins-config-volume\") pod \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\" (UID: \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\") "
Oct 30 13:41:57.015394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.015340    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-kube-api-access-lmbw7\" (UniqueName: \"kubernetes.io/projected/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-kube-api-access-lmbw7\") pod \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\" (UID: \"9ad53602-bf0d-4ca4-9261-e515e8b05e4e\") "
Oct 30 13:41:57.015527 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.015502    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-plugins-config-volume" pod "9ad53602-bf0d-4ca4-9261-e515e8b05e4e" (UID: "9ad53602-bf0d-4ca4-9261-e515e8b05e4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:57.016171 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.016146    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:57.017823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.017796    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-kube-api-access-lmbw7" pod "9ad53602-bf0d-4ca4-9261-e515e8b05e4e" (UID: "9ad53602-bf0d-4ca4-9261-e515e8b05e4e"). InnerVolumeSpecName "kube-api-access-lmbw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:57.021102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.021083    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:41:57.117068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.117040    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f35f65b-2202-4bd9-b8ea-705c55d37960-kube-api-access-vt8jb\" (UniqueName: \"kubernetes.io/projected/4f35f65b-2202-4bd9-b8ea-705c55d37960-kube-api-access-vt8jb\") pod \"4f35f65b-2202-4bd9-b8ea-705c55d37960\" (UID: \"4f35f65b-2202-4bd9-b8ea-705c55d37960\") "
Oct 30 13:41:57.117439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.117421    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f35f65b-2202-4bd9-b8ea-705c55d37960-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f35f65b-2202-4bd9-b8ea-705c55d37960-plugins-config-volume\") pod \"4f35f65b-2202-4bd9-b8ea-705c55d37960\" (UID: \"4f35f65b-2202-4bd9-b8ea-705c55d37960\") "
Oct 30 13:41:57.117807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.117779    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f35f65b-2202-4bd9-b8ea-705c55d37960-plugins-config-volume" pod "4f35f65b-2202-4bd9-b8ea-705c55d37960" (UID: "4f35f65b-2202-4bd9-b8ea-705c55d37960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:57.118148 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.118129    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbw7\" (UniqueName: \"kubernetes.io/projected/9ad53602-bf0d-4ca4-9261-e515e8b05e4e-kube-api-access-lmbw7\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:57.118232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.118154    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f35f65b-2202-4bd9-b8ea-705c55d37960-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:57.123443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.123412    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f35f65b-2202-4bd9-b8ea-705c55d37960-kube-api-access-vt8jb" pod "4f35f65b-2202-4bd9-b8ea-705c55d37960" (UID: "4f35f65b-2202-4bd9-b8ea-705c55d37960"). InnerVolumeSpecName "kube-api-access-vt8jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:57.220099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.220064    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt8jb\" (UniqueName: \"kubernetes.io/projected/4f35f65b-2202-4bd9-b8ea-705c55d37960-kube-api-access-vt8jb\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:41:57.323806 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.323770    4429 generic.go:356] "Generic (PLEG): container finished" podID="9ad53602-bf0d-4ca4-9261-e515e8b05e4e" containerID="29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3" exitCode=0
Oct 30 13:41:57.323983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.323819    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"
Oct 30 13:41:57.323983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.323858    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg" event={"ID":"9ad53602-bf0d-4ca4-9261-e515e8b05e4e","Type":"ContainerDied","Data":"29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3"}
Oct 30 13:41:57.323983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.323911    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg" event={"ID":"9ad53602-bf0d-4ca4-9261-e515e8b05e4e","Type":"ContainerDied","Data":"1c289d3bc5252a9010d9ab01dacb278ba1c3e020c0f2f0a1724340ae977b524d"}
Oct 30 13:41:57.323983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.323932    4429 scope.go:122] "RemoveContainer" containerID="29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3"
Oct 30 13:41:57.325544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.325524    4429 generic.go:356] "Generic (PLEG): container finished" podID="4f35f65b-2202-4bd9-b8ea-705c55d37960" containerID="c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1" exitCode=0
Oct 30 13:41:57.325676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.325642    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"
Oct 30 13:41:57.325728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.325689    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d" event={"ID":"4f35f65b-2202-4bd9-b8ea-705c55d37960","Type":"ContainerDied","Data":"c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1"}
Oct 30 13:41:57.325773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.325732    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d" event={"ID":"4f35f65b-2202-4bd9-b8ea-705c55d37960","Type":"ContainerDied","Data":"891bc1ec8c5d9b30f6345c4e7c57bfb566ad614793dfeaa3e58180c7f302274b"}
Oct 30 13:41:57.335588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.335561    4429 scope.go:122] "RemoveContainer" containerID="29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3"
Oct 30 13:41:57.335842 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:41:57.335819    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3\": container with ID starting with 29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3 not found: ID does not exist" containerID="29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3"
Oct 30 13:41:57.335893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.335846    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3"} err="failed to get container status \"29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3\": rpc error: code = NotFound desc = could not find container \"29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3\": container with ID starting with 29599d30a95a788a22d02fd87806d29debbfb5a4d06add599a794f5a0c7acce3 not found: ID does not exist"
Oct 30 13:41:57.335893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.335865    4429 scope.go:122] "RemoveContainer" containerID="c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1"
Oct 30 13:41:57.347740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.347723    4429 scope.go:122] "RemoveContainer" containerID="c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1"
Oct 30 13:41:57.348002 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:41:57.347984    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1\": container with ID starting with c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1 not found: ID does not exist" containerID="c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1"
Oct 30 13:41:57.348072 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.348009    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1"} err="failed to get container status \"c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1\": rpc error: code = NotFound desc = could not find container \"c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1\": container with ID starting with c4e9e45515033d1c653e815ed2908ae085914b047c716ed8e6c3137bffcc27e1 not found: ID does not exist"
Oct 30 13:41:57.402355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.402322    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"]
Oct 30 13:41:57.416938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.416900    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d41320-6fbcbd98fb-8kmvg"]
Oct 30 13:41:57.454979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.454932    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"]
Oct 30 13:41:57.460690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:57.460668    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d41320-687fb8bd99-pz64d"]
Oct 30 13:41:58.078198 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:58.078166    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f35f65b-2202-4bd9-b8ea-705c55d37960" path="/var/lib/kubelet/pods/4f35f65b-2202-4bd9-b8ea-705c55d37960/volumes"
Oct 30 13:41:58.078640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:41:58.078625    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad53602-bf0d-4ca4-9261-e515e8b05e4e" path="/var/lib/kubelet/pods/9ad53602-bf0d-4ca4-9261-e515e8b05e4e/volumes"
Oct 30 13:42:04.313326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.313224    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"]
Oct 30 13:42:04.313823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.313441    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7" podUID="ef3af666-4cad-4951-8284-0daf00f7eb3d" containerName="maas-api" containerID="cri-o://4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b" gracePeriod=30
Oct 30 13:42:04.435324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.435261    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7" podUID="ef3af666-4cad-4951-8284-0daf00f7eb3d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.219:8443/health\": dial tcp 10.133.0.219:8443: connect: connection refused"
Oct 30 13:42:04.672343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.672303    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:42:04.854892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.854859    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-maas-api-tls\") pod \"ef3af666-4cad-4951-8284-0daf00f7eb3d\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") "
Oct 30 13:42:04.855194 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.855177    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-metrics-certs\") pod \"ef3af666-4cad-4951-8284-0daf00f7eb3d\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") "
Oct 30 13:42:04.855297 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.855215    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3af666-4cad-4951-8284-0daf00f7eb3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3af666-4cad-4951-8284-0daf00f7eb3d-openshift-service-ca\") pod \"ef3af666-4cad-4951-8284-0daf00f7eb3d\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") "
Oct 30 13:42:04.855379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.855365    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3af666-4cad-4951-8284-0daf00f7eb3d-kube-api-access-kzsbp\" (UniqueName: \"kubernetes.io/projected/ef3af666-4cad-4951-8284-0daf00f7eb3d-kube-api-access-kzsbp\") pod \"ef3af666-4cad-4951-8284-0daf00f7eb3d\" (UID: \"ef3af666-4cad-4951-8284-0daf00f7eb3d\") "
Oct 30 13:42:04.855759 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.855730    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3af666-4cad-4951-8284-0daf00f7eb3d-openshift-service-ca" pod "ef3af666-4cad-4951-8284-0daf00f7eb3d" (UID: "ef3af666-4cad-4951-8284-0daf00f7eb3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:04.856188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.856164    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3af666-4cad-4951-8284-0daf00f7eb3d-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:42:04.857465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.857434    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-maas-api-tls" pod "ef3af666-4cad-4951-8284-0daf00f7eb3d" (UID: "ef3af666-4cad-4951-8284-0daf00f7eb3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:04.857974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.857948    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3af666-4cad-4951-8284-0daf00f7eb3d-kube-api-access-kzsbp" pod "ef3af666-4cad-4951-8284-0daf00f7eb3d" (UID: "ef3af666-4cad-4951-8284-0daf00f7eb3d"). InnerVolumeSpecName "kube-api-access-kzsbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:04.858666 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.858642    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-metrics-certs" pod "ef3af666-4cad-4951-8284-0daf00f7eb3d" (UID: "ef3af666-4cad-4951-8284-0daf00f7eb3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:04.958410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.958363    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:42:04.958410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.958404    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzsbp\" (UniqueName: \"kubernetes.io/projected/ef3af666-4cad-4951-8284-0daf00f7eb3d-kube-api-access-kzsbp\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:42:04.958649 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:04.958422    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef3af666-4cad-4951-8284-0daf00f7eb3d-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:42:05.395962 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.395926    4429 generic.go:356] "Generic (PLEG): container finished" podID="ef3af666-4cad-4951-8284-0daf00f7eb3d" containerID="4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b" exitCode=0
Oct 30 13:42:05.396416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.395992    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"
Oct 30 13:42:05.396416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.396012    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7" event={"ID":"ef3af666-4cad-4951-8284-0daf00f7eb3d","Type":"ContainerDied","Data":"4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b"}
Oct 30 13:42:05.396416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.396111    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7" event={"ID":"ef3af666-4cad-4951-8284-0daf00f7eb3d","Type":"ContainerDied","Data":"8717628a9e731343e0e59cf27c3d139dba952d8f1e0107c40a283a7962407e3e"}
Oct 30 13:42:05.396416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.396140    4429 scope.go:122] "RemoveContainer" containerID="4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b"
Oct 30 13:42:05.409053 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.409029    4429 scope.go:122] "RemoveContainer" containerID="4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b"
Oct 30 13:42:05.409542 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:42:05.409516    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b\": container with ID starting with 4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b not found: ID does not exist" containerID="4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b"
Oct 30 13:42:05.409641 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.409551    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b"} err="failed to get container status \"4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b\": rpc error: code = NotFound desc = could not find container \"4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b\": container with ID starting with 4b5ff9b6db745538aac88dec46d9c3a9a9ff7e4c410be82619d69e0af122619b not found: ID does not exist"
Oct 30 13:42:05.459893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.459860    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"]
Oct 30 13:42:05.475123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:05.475093    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d41320-79fb5b7c8c-486j7"]
Oct 30 13:42:06.079559 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:42:06.079521    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3af666-4cad-4951-8284-0daf00f7eb3d" path="/var/lib/kubelet/pods/ef3af666-4cad-4951-8284-0daf00f7eb3d/volumes"
Oct 30 13:42:12.373133 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:42:12.373094    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:42:42.375663 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:42:42.375626    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:43:12.376011 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:43:12.375974    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:43:15.504277 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:15.504241    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"]
Oct 30 13:43:15.504702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:15.504497    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy" containerID="cri-o://0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea" gracePeriod=30
Oct 30 13:43:20.808159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.808136    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:43:20.898034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.897998    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-credential-socket\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.898237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898067    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-kube-api-access-hqtpc\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-kube-api-access-hqtpc\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.898237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898107    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-certs\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.898237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898148    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-socket\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.898510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898238    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16ce256-b633-48e7-9679-772bc5b6fee7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ce256-b633-48e7-9679-772bc5b6fee7-istiod-ca-cert\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.898510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898334    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-credential-socket" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.898510 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898297    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-podinfo\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.898684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898590    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-socket" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.898744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898715    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-certs" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.898803 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898751    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16ce256-b633-48e7-9679-772bc5b6fee7-istiod-ca-cert" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:20.898867 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.898832    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-envoy\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.899041 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.899024    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-token\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-token\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.899122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.899086    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-data\") pod \"c16ce256-b633-48e7-9679-772bc5b6fee7\" (UID: \"c16ce256-b633-48e7-9679-772bc5b6fee7\") "
Oct 30 13:43:20.900152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.899464    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-data" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:20.900152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.900056    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.900152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.900078    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ce256-b633-48e7-9679-772bc5b6fee7-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.900152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.900092    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.900152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.900105    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.900152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.900118    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:20.902732 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.902612    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-kube-api-access-hqtpc" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "kube-api-access-hqtpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:20.902732 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.902703    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-podinfo" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:20.902977 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.902952    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-token" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:20.904347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:20.904322    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-envoy" pod "c16ce256-b633-48e7-9679-772bc5b6fee7" (UID: "c16ce256-b633-48e7-9679-772bc5b6fee7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:21.009954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.009914    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.009954 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.009957    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.017119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.009972    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.017119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.009996    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqtpc\" (UniqueName: \"kubernetes.io/projected/c16ce256-b633-48e7-9679-772bc5b6fee7-kube-api-access-hqtpc\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:21.113826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.113739    4429 generic.go:356] "Generic (PLEG): container finished" podID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerID="0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea" exitCode=0
Oct 30 13:43:21.113826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.113806    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"
Oct 30 13:43:21.114073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.113819    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" event={"ID":"c16ce256-b633-48e7-9679-772bc5b6fee7","Type":"ContainerDied","Data":"0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea"}
Oct 30 13:43:21.114073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.113878    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j" event={"ID":"c16ce256-b633-48e7-9679-772bc5b6fee7","Type":"ContainerDied","Data":"f0783a6c2af96f0fd218aeded312080b19dec16f146df211058078ec3c3f6f27"}
Oct 30 13:43:21.114073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.113910    4429 scope.go:122] "RemoveContainer" containerID="0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea"
Oct 30 13:43:21.126794 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.126772    4429 scope.go:122] "RemoveContainer" containerID="0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea"
Oct 30 13:43:21.127095 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:43:21.127075    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea\": container with ID starting with 0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea not found: ID does not exist" containerID="0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea"
Oct 30 13:43:21.127177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.127108    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea"} err="failed to get container status \"0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea\": rpc error: code = NotFound desc = could not find container \"0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea\": container with ID starting with 0bba3cea701c31a214709de3d31126751f5884cec79eb64f0d7dc3e69c86b0ea not found: ID does not exist"
Oct 30 13:43:21.178558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.178530    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"]
Oct 30 13:43:21.192003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:21.191976    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-77b6j"]
Oct 30 13:43:22.081150 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:22.081112    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" path="/var/lib/kubelet/pods/c16ce256-b633-48e7-9679-772bc5b6fee7/volumes"
Oct 30 13:43:42.373088 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:43:42.373009    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:43:49.825319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:49.825264    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"]
Oct 30 13:43:49.825730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:49.825488    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy" containerID="cri-o://cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe" gracePeriod=30
Oct 30 13:43:55.114797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.114768    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:43:55.158455 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158430    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-credential-socket\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.158594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158469    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-kube-api-access-fr4hk\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-kube-api-access-fr4hk\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.158594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158521    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-certs\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.158594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158572    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-socket\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.158749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158680    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7908707e-27d0-47a1-9633-69f77500fc76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7908707e-27d0-47a1-9633-69f77500fc76-istiod-ca-cert\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.158749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158705    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-data\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.158749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158719    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-credential-socket" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:55.158922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158732    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7908707e-27d0-47a1-9633-69f77500fc76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7908707e-27d0-47a1-9633-69f77500fc76-istio-podinfo\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.158922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158825    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-certs" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:55.158922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158902    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-envoy\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.159070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.158929    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-socket" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:55.159070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.159002    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-istio-token\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-istio-token\") pod \"7908707e-27d0-47a1-9633-69f77500fc76\" (UID: \"7908707e-27d0-47a1-9633-69f77500fc76\") "
Oct 30 13:43:55.159070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.159014    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-data" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:55.159250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.159118    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7908707e-27d0-47a1-9633-69f77500fc76-istiod-ca-cert" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:55.160521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.160485    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.160521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.160513    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.160688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.160528    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.160688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.160541    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7908707e-27d0-47a1-9633-69f77500fc76-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.160688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.160572    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.161105 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.161084    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-kube-api-access-fr4hk" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "kube-api-access-fr4hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:55.161327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.161273    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7908707e-27d0-47a1-9633-69f77500fc76-istio-podinfo" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:43:55.161423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.161396    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-envoy" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:55.161574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.161555    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-istio-token" pod "7908707e-27d0-47a1-9633-69f77500fc76" (UID: "7908707e-27d0-47a1-9633-69f77500fc76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:55.262330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.262300    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.262330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.262327    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr4hk\" (UniqueName: \"kubernetes.io/projected/7908707e-27d0-47a1-9633-69f77500fc76-kube-api-access-fr4hk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.262330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.262337    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7908707e-27d0-47a1-9633-69f77500fc76-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.262557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.262347    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7908707e-27d0-47a1-9633-69f77500fc76-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:43:55.419944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.419864    4429 generic.go:356] "Generic (PLEG): container finished" podID="7908707e-27d0-47a1-9633-69f77500fc76" containerID="cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe" exitCode=0
Oct 30 13:43:55.419944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.419906    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" event={"ID":"7908707e-27d0-47a1-9633-69f77500fc76","Type":"ContainerDied","Data":"cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe"}
Oct 30 13:43:55.419944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.419921    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"
Oct 30 13:43:55.420207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.419947    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj" event={"ID":"7908707e-27d0-47a1-9633-69f77500fc76","Type":"ContainerDied","Data":"1e81e05564bdf87015a0bd4935b06a121b48be10a2aa47838396bb14e28807bb"}
Oct 30 13:43:55.420207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.419969    4429 scope.go:122] "RemoveContainer" containerID="cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe"
Oct 30 13:43:55.441412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.441391    4429 scope.go:122] "RemoveContainer" containerID="cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe"
Oct 30 13:43:55.441779 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:43:55.441743    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe\": container with ID starting with cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe not found: ID does not exist" containerID="cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe"
Oct 30 13:43:55.441862 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.441793    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe"} err="failed to get container status \"cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe\": rpc error: code = NotFound desc = could not find container \"cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe\": container with ID starting with cb2646c0304d8835d1f392ddea9b4614887630e0d58f72352e089e11ea5e42fe not found: ID does not exist"
Oct 30 13:43:55.489912 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.489876    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"]
Oct 30 13:43:55.499079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:55.499040    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9a93bd-openshift-default-6bbb8569cc-hmhgj"]
Oct 30 13:43:56.081875 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:43:56.081784    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7908707e-27d0-47a1-9633-69f77500fc76" path="/var/lib/kubelet/pods/7908707e-27d0-47a1-9633-69f77500fc76/volumes"
Oct 30 13:44:01.633609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:01.633578    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"]
Oct 30 13:44:12.373314 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:44:12.373248    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:44:16.655886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:16.655851    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj" podUID="dad2b06f-919b-40e7-bb40-259fc11e2b97" containerName="main" containerID="cri-o://90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676" gracePeriod=45
Oct 30 13:44:16.965888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:16.965863    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:44:17.092217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092182    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad2b06f-919b-40e7-bb40-259fc11e2b97-kube-api-access-5fwtn\" (UniqueName: \"kubernetes.io/projected/dad2b06f-919b-40e7-bb40-259fc11e2b97-kube-api-access-5fwtn\") pod \"dad2b06f-919b-40e7-bb40-259fc11e2b97\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") "
Oct 30 13:44:17.092409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092276    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-home\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-home\") pod \"dad2b06f-919b-40e7-bb40-259fc11e2b97\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") "
Oct 30 13:44:17.092472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092418    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-model-cache\") pod \"dad2b06f-919b-40e7-bb40-259fc11e2b97\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") "
Oct 30 13:44:17.092561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092543    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad2b06f-919b-40e7-bb40-259fc11e2b97-tls-certs\" (UniqueName: \"kubernetes.io/secret/dad2b06f-919b-40e7-bb40-259fc11e2b97-tls-certs\") pod \"dad2b06f-919b-40e7-bb40-259fc11e2b97\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") "
Oct 30 13:44:17.092605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092581    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-home" pod "dad2b06f-919b-40e7-bb40-259fc11e2b97" (UID: "dad2b06f-919b-40e7-bb40-259fc11e2b97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:17.092651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092618    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-dshm\") pod \"dad2b06f-919b-40e7-bb40-259fc11e2b97\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") "
Oct 30 13:44:17.092720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092651    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-tmp-dir\") pod \"dad2b06f-919b-40e7-bb40-259fc11e2b97\" (UID: \"dad2b06f-919b-40e7-bb40-259fc11e2b97\") "
Oct 30 13:44:17.092720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092679    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-model-cache" pod "dad2b06f-919b-40e7-bb40-259fc11e2b97" (UID: "dad2b06f-919b-40e7-bb40-259fc11e2b97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:17.093045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.092955    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-tmp-dir" pod "dad2b06f-919b-40e7-bb40-259fc11e2b97" (UID: "dad2b06f-919b-40e7-bb40-259fc11e2b97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:17.093255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.093237    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:17.093346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.093261    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:17.093346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.093274    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:17.095912 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.095878    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad2b06f-919b-40e7-bb40-259fc11e2b97-tls-certs" pod "dad2b06f-919b-40e7-bb40-259fc11e2b97" (UID: "dad2b06f-919b-40e7-bb40-259fc11e2b97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:17.095912 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.095899    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad2b06f-919b-40e7-bb40-259fc11e2b97-kube-api-access-5fwtn" pod "dad2b06f-919b-40e7-bb40-259fc11e2b97" (UID: "dad2b06f-919b-40e7-bb40-259fc11e2b97"). InnerVolumeSpecName "kube-api-access-5fwtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:17.096086 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.095942    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-dshm" pod "dad2b06f-919b-40e7-bb40-259fc11e2b97" (UID: "dad2b06f-919b-40e7-bb40-259fc11e2b97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:17.194969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.194943    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad2b06f-919b-40e7-bb40-259fc11e2b97-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:17.195095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.194972    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad2b06f-919b-40e7-bb40-259fc11e2b97-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:17.195095 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.194990    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fwtn\" (UniqueName: \"kubernetes.io/projected/dad2b06f-919b-40e7-bb40-259fc11e2b97-kube-api-access-5fwtn\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:17.604832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.604735    4429 generic.go:356] "Generic (PLEG): container finished" podID="dad2b06f-919b-40e7-bb40-259fc11e2b97" containerID="90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676" exitCode=0
Oct 30 13:44:17.604832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.604794    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"
Oct 30 13:44:17.604832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.604794    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj" event={"ID":"dad2b06f-919b-40e7-bb40-259fc11e2b97","Type":"ContainerDied","Data":"90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676"}
Oct 30 13:44:17.605125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.604846    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj" event={"ID":"dad2b06f-919b-40e7-bb40-259fc11e2b97","Type":"ContainerDied","Data":"a87a00cc102825a4e59d522aa76d31f1e572a9b7485f722999be02433ba8753b"}
Oct 30 13:44:17.605125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.604866    4429 scope.go:122] "RemoveContainer" containerID="90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676"
Oct 30 13:44:17.618862 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.618845    4429 scope.go:122] "RemoveContainer" containerID="90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676"
Oct 30 13:44:17.619244 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:44:17.619221    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676\": container with ID starting with 90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676 not found: ID does not exist" containerID="90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676"
Oct 30 13:44:17.619409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.619248    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676"} err="failed to get container status \"90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676\": rpc error: code = NotFound desc = could not find container \"90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676\": container with ID starting with 90330cec499a241a94a5e95cba4f68d1eecfa2e997e989e4e81e8f29759e8676 not found: ID does not exist"
Oct 30 13:44:17.663794 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.663766    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"]
Oct 30 13:44:17.677217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:17.677189    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-22efcf/test-model-22efcf-kserve-5bbdcc56f9-stkzj"]
Oct 30 13:44:18.077654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:18.077622    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad2b06f-919b-40e7-bb40-259fc11e2b97" path="/var/lib/kubelet/pods/dad2b06f-919b-40e7-bb40-259fc11e2b97/volumes"
Oct 30 13:44:21.560609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:21.560573    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"]
Oct 30 13:44:22.156153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:22.156121    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"]
Oct 30 13:44:30.740174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.740142    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"]
Oct 30 13:44:30.741197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741173    4429 cpu_manager.go:414] "containerMap: removing container" podUID="4f35f65b-2202-4bd9-b8ea-705c55d37960" containerName="payload-pre-processing"
Oct 30 13:44:30.741197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741196    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f35f65b-2202-4bd9-b8ea-705c55d37960" containerName="payload-pre-processing"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741234    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9ad53602-bf0d-4ca4-9261-e515e8b05e4e" containerName="payload-processing"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741244    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad53602-bf0d-4ca4-9261-e515e8b05e4e" containerName="payload-processing"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741262    4429 cpu_manager.go:414] "containerMap: removing container" podUID="dad2b06f-919b-40e7-bb40-259fc11e2b97" containerName="main"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741269    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad2b06f-919b-40e7-bb40-259fc11e2b97" containerName="main"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741312    4429 cpu_manager.go:414] "containerMap: removing container" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741319    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741341    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741348    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741385    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ef3af666-4cad-4951-8284-0daf00f7eb3d" containerName="maas-api"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741390    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3af666-4cad-4951-8284-0daf00f7eb3d" containerName="maas-api"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741526    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad53602-bf0d-4ca4-9261-e515e8b05e4e" containerName="payload-processing"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741535    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad2b06f-919b-40e7-bb40-259fc11e2b97" containerName="main"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741546    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="7908707e-27d0-47a1-9633-69f77500fc76" containerName="istio-proxy"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741553    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16ce256-b633-48e7-9679-772bc5b6fee7" containerName="istio-proxy"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741563    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3af666-4cad-4951-8284-0daf00f7eb3d" containerName="maas-api"
Oct 30 13:44:30.741668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.741569    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f35f65b-2202-4bd9-b8ea-705c55d37960" containerName="payload-pre-processing"
Oct 30 13:44:30.746262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.746245    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.776625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.776573    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.776801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.776715    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.776884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.776808    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.777014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.776990    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.777116 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.777098    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.777173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.777138    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.777247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.777226    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.777376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.777354    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.777449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.777410    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9zg\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-kube-api-access-dq9zg\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.818326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.783666    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"]
Oct 30 13:44:30.878767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.878736    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.878958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.878791    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.878958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.878813    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.878958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.878843    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.878958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.878892    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.878958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.878931    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9zg\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-kube-api-access-dq9zg\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.879244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.879072    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.879244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.879141    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.879244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.879219    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.879471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.879249    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.879471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.879349    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.879585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.879539    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.879585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.879554    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.879872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.879784    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.881206 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.881179    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.881513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.881493    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.904807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.904777    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:30.904931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:30.904879    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9zg\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-kube-api-access-dq9zg\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7hbnd\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:31.073919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:31.073846    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:31.397991 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:44:31.397959    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64ff59a_c76d_41ab_9c2a_55c717e626cf.slice/crio-9f0c21363e9dcfc9c4657329046b9e42d25945a72f0f04ccfc73443f38f250f8 WatchSource:0}: Error finding container 9f0c21363e9dcfc9c4657329046b9e42d25945a72f0f04ccfc73443f38f250f8: Status 404 returned error can't find the container with id 9f0c21363e9dcfc9c4657329046b9e42d25945a72f0f04ccfc73443f38f250f8
Oct 30 13:44:31.399609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:31.399589    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"]
Oct 30 13:44:31.400835 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:31.400803    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:44:31.400893 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:31.400872    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:44:31.400929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:31.400902    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:44:31.734851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:31.734817    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" event={"ID":"e64ff59a-c76d-41ab-9c2a-55c717e626cf","Type":"ContainerStarted","Data":"752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f"}
Oct 30 13:44:31.735042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:31.734877    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" event={"ID":"e64ff59a-c76d-41ab-9c2a-55c717e626cf","Type":"ContainerStarted","Data":"9f0c21363e9dcfc9c4657329046b9e42d25945a72f0f04ccfc73443f38f250f8"}
Oct 30 13:44:31.801794 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:31.801737    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podStartSLOduration=1.80171938 podStartE2EDuration="1.80171938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:31.794361936 +0000 UTC m=+4329.849855760" watchObservedRunningTime="2026-10-30 13:44:31.80171938 +0000 UTC m=+4329.857213206"
Oct 30 13:44:32.076232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:32.076160    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7hbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:44:32.076232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:32.076206    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:44:32.077589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:32.077571    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:33.090309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:33.089773    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7hbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:44:33.090309 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:33.089831    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:44:34.074273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:34.074235    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7hbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:44:34.074505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:34.074331    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:44:35.074766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:35.074729    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7hbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:44:35.075160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:35.074789    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:44:36.075886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:36.075795    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7hbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:44:36.075886 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:36.075872    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:44:36.585054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:36.585017    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx" podUID="a57ec178-409a-489c-bf08-0da9d6db5da5" containerName="main" containerID="cri-o://d4e39c78b981d5fd9de496dacb23f57dd4f9f98146755791ff713a451ae02237" gracePeriod=45
Oct 30 13:44:36.831860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:36.831605    4429 generic.go:356] "Generic (PLEG): container finished" podID="a57ec178-409a-489c-bf08-0da9d6db5da5" containerID="d4e39c78b981d5fd9de496dacb23f57dd4f9f98146755791ff713a451ae02237" exitCode=0
Oct 30 13:44:36.831860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:36.831816    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx" event={"ID":"a57ec178-409a-489c-bf08-0da9d6db5da5","Type":"ContainerDied","Data":"d4e39c78b981d5fd9de496dacb23f57dd4f9f98146755791ff713a451ae02237"}
Oct 30 13:44:36.919472 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:36.919446    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:44:37.010346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.010271    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57ec178-409a-489c-bf08-0da9d6db5da5-kube-api-access-ssc4v\" (UniqueName: \"kubernetes.io/projected/a57ec178-409a-489c-bf08-0da9d6db5da5-kube-api-access-ssc4v\") pod \"a57ec178-409a-489c-bf08-0da9d6db5da5\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") "
Oct 30 13:44:37.010539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.010440    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-dshm\") pod \"a57ec178-409a-489c-bf08-0da9d6db5da5\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") "
Oct 30 13:44:37.010539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.010470    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-model-cache\") pod \"a57ec178-409a-489c-bf08-0da9d6db5da5\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") "
Oct 30 13:44:37.010539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.010524    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-home\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-home\") pod \"a57ec178-409a-489c-bf08-0da9d6db5da5\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") "
Oct 30 13:44:37.010724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.010697    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a57ec178-409a-489c-bf08-0da9d6db5da5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a57ec178-409a-489c-bf08-0da9d6db5da5-tls-certs\") pod \"a57ec178-409a-489c-bf08-0da9d6db5da5\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") "
Oct 30 13:44:37.010801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.010757    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-model-cache" pod "a57ec178-409a-489c-bf08-0da9d6db5da5" (UID: "a57ec178-409a-489c-bf08-0da9d6db5da5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:37.010863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.010825    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-tmp-dir\") pod \"a57ec178-409a-489c-bf08-0da9d6db5da5\" (UID: \"a57ec178-409a-489c-bf08-0da9d6db5da5\") "
Oct 30 13:44:37.010950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.010922    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-home" pod "a57ec178-409a-489c-bf08-0da9d6db5da5" (UID: "a57ec178-409a-489c-bf08-0da9d6db5da5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:37.011518 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.011486    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-tmp-dir" pod "a57ec178-409a-489c-bf08-0da9d6db5da5" (UID: "a57ec178-409a-489c-bf08-0da9d6db5da5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:37.011632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.011581    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.011632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.011599    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.014227 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.014199    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-dshm" pod "a57ec178-409a-489c-bf08-0da9d6db5da5" (UID: "a57ec178-409a-489c-bf08-0da9d6db5da5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:37.014355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.014332    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57ec178-409a-489c-bf08-0da9d6db5da5-tls-certs" pod "a57ec178-409a-489c-bf08-0da9d6db5da5" (UID: "a57ec178-409a-489c-bf08-0da9d6db5da5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:37.014407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.014385    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57ec178-409a-489c-bf08-0da9d6db5da5-kube-api-access-ssc4v" pod "a57ec178-409a-489c-bf08-0da9d6db5da5" (UID: "a57ec178-409a-489c-bf08-0da9d6db5da5"). InnerVolumeSpecName "kube-api-access-ssc4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:37.074162 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.074122    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7hbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:44:37.074357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.074176    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:44:37.113070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.113032    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssc4v\" (UniqueName: \"kubernetes.io/projected/a57ec178-409a-489c-bf08-0da9d6db5da5-kube-api-access-ssc4v\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.113070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.113067    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.113511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.113082    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a57ec178-409a-489c-bf08-0da9d6db5da5-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.113511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.113097    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57ec178-409a-489c-bf08-0da9d6db5da5-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.180921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.180813    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp" podUID="21b02805-60fc-4eee-9f57-52e1d7e1d6b7" containerName="main" containerID="cri-o://db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e" gracePeriod=45
Oct 30 13:44:37.492172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.492137    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:44:37.622261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.622224    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-kube-api-access-kt2g9\" (UniqueName: \"kubernetes.io/projected/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-kube-api-access-kt2g9\") pod \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") "
Oct 30 13:44:37.622522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.622504    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tmp-dir\") pod \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") "
Oct 30 13:44:37.622592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.622554    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tls-certs\") pod \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") "
Oct 30 13:44:37.622592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.622577    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-dshm\") pod \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") "
Oct 30 13:44:37.622683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.622670    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-model-cache\") pod \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") "
Oct 30 13:44:37.622724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.622687    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-home\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-home\") pod \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\" (UID: \"21b02805-60fc-4eee-9f57-52e1d7e1d6b7\") "
Oct 30 13:44:37.623393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.623367    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-home" pod "21b02805-60fc-4eee-9f57-52e1d7e1d6b7" (UID: "21b02805-60fc-4eee-9f57-52e1d7e1d6b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:37.623708 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.623688    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-model-cache" pod "21b02805-60fc-4eee-9f57-52e1d7e1d6b7" (UID: "21b02805-60fc-4eee-9f57-52e1d7e1d6b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:37.623797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.623707    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tmp-dir" pod "21b02805-60fc-4eee-9f57-52e1d7e1d6b7" (UID: "21b02805-60fc-4eee-9f57-52e1d7e1d6b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:37.626171 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.626136    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-dshm" pod "21b02805-60fc-4eee-9f57-52e1d7e1d6b7" (UID: "21b02805-60fc-4eee-9f57-52e1d7e1d6b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:44:37.626445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.626423    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-kube-api-access-kt2g9" pod "21b02805-60fc-4eee-9f57-52e1d7e1d6b7" (UID: "21b02805-60fc-4eee-9f57-52e1d7e1d6b7"). InnerVolumeSpecName "kube-api-access-kt2g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:37.627424 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.627397    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tls-certs" pod "21b02805-60fc-4eee-9f57-52e1d7e1d6b7" (UID: "21b02805-60fc-4eee-9f57-52e1d7e1d6b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:37.725488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.725452    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.725488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.725490    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.725728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.725505    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2g9\" (UniqueName: \"kubernetes.io/projected/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-kube-api-access-kt2g9\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.725728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.725522    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.725728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.725539    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.725728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.725551    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b02805-60fc-4eee-9f57-52e1d7e1d6b7-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:44:37.844130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.844039    4429 generic.go:356] "Generic (PLEG): container finished" podID="21b02805-60fc-4eee-9f57-52e1d7e1d6b7" containerID="db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e" exitCode=0
Oct 30 13:44:37.844130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.844083    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"
Oct 30 13:44:37.844388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.844130    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp" event={"ID":"21b02805-60fc-4eee-9f57-52e1d7e1d6b7","Type":"ContainerDied","Data":"db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e"}
Oct 30 13:44:37.844388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.844171    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp" event={"ID":"21b02805-60fc-4eee-9f57-52e1d7e1d6b7","Type":"ContainerDied","Data":"4cce6958a43ffb76ce1328ab22032ce8a11c925c97360cc2353ec0cfde96569e"}
Oct 30 13:44:37.844388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.844192    4429 scope.go:122] "RemoveContainer" containerID="db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e"
Oct 30 13:44:37.846034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.846008    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx" event={"ID":"a57ec178-409a-489c-bf08-0da9d6db5da5","Type":"ContainerDied","Data":"eb854d70e01f0040a59f4684c70e66523f5ce1efb35f22399f4fb2e8cd96ca86"}
Oct 30 13:44:37.846034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.846033    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"
Oct 30 13:44:37.860858 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.860822    4429 scope.go:122] "RemoveContainer" containerID="db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e"
Oct 30 13:44:37.861538 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:44:37.861357    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e\": container with ID starting with db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e not found: ID does not exist" containerID="db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e"
Oct 30 13:44:37.861538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.861392    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e"} err="failed to get container status \"db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e\": rpc error: code = NotFound desc = could not find container \"db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e\": container with ID starting with db7b8aeef251435eac267683a8605b6ee1a2ab4aa96fa037aff0fee6bd4cc42e not found: ID does not exist"
Oct 30 13:44:37.861538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.861416    4429 scope.go:122] "RemoveContainer" containerID="d4e39c78b981d5fd9de496dacb23f57dd4f9f98146755791ff713a451ae02237"
Oct 30 13:44:37.923492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.923461    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"]
Oct 30 13:44:37.936372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.936338    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9cc0e6/test-model-9cc0e6-kserve-596f878b59-8dfqx"]
Oct 30 13:44:37.971677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.971649    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"]
Oct 30 13:44:37.975915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:37.975887    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7b7407/test-model-7b7407-kserve-5db7c886cc-c5snp"]
Oct 30 13:44:38.074931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:38.074904    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7hbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:44:38.075132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:38.074954    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:44:38.084418 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:38.084378    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b02805-60fc-4eee-9f57-52e1d7e1d6b7" path="/var/lib/kubelet/pods/21b02805-60fc-4eee-9f57-52e1d7e1d6b7/volumes"
Oct 30 13:44:38.085314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:38.085273    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57ec178-409a-489c-bf08-0da9d6db5da5" path="/var/lib/kubelet/pods/a57ec178-409a-489c-bf08-0da9d6db5da5/volumes"
Oct 30 13:44:39.074230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:39.074194    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7hbnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 30 13:44:39.074631 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:39.074261    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 30 13:44:40.115533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:40.115502    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:40.116009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:40.115986    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:40.117221 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:40.117159    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:44:42.375528 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:44:42.375491    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:44:45.784801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.784764    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"]
Oct 30 13:44:45.785823 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.785801    4429 cpu_manager.go:414] "containerMap: removing container" podUID="21b02805-60fc-4eee-9f57-52e1d7e1d6b7" containerName="main"
Oct 30 13:44:45.785979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.785957    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b02805-60fc-4eee-9f57-52e1d7e1d6b7" containerName="main"
Oct 30 13:44:45.786071 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.786016    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a57ec178-409a-489c-bf08-0da9d6db5da5" containerName="main"
Oct 30 13:44:45.786071 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.786021    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57ec178-409a-489c-bf08-0da9d6db5da5" containerName="main"
Oct 30 13:44:45.786200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.786190    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57ec178-409a-489c-bf08-0da9d6db5da5" containerName="main"
Oct 30 13:44:45.786235 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.786214    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b02805-60fc-4eee-9f57-52e1d7e1d6b7" containerName="main"
Oct 30 13:44:45.791814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.791798    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.815242 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.815216    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"]
Oct 30 13:44:45.953892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.953862    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.954052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.953985    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.954052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.954021    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.954133 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.954124    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e886c2e-e044-4f32-af32-9945c531b70f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.954173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.954139    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.954232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.954211    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.954278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.954267    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.954394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.954380    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88f8\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-kube-api-access-j88f8\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:45.954431 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:45.954419    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e886c2e-e044-4f32-af32-9945c531b70f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055170    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e886c2e-e044-4f32-af32-9945c531b70f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055225    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055318    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055347    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055427    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e886c2e-e044-4f32-af32-9945c531b70f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055440    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055473    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055529    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055624    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88f8\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-kube-api-access-j88f8\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055699    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055725    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055822    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055963    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.055989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.055970    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e886c2e-e044-4f32-af32-9945c531b70f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.057877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.057855    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.057991 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.057973    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e886c2e-e044-4f32-af32-9945c531b70f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.080753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.080726    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88f8\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-kube-api-access-j88f8\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.080753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.080748    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-xpssg\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.107163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.107144    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:46.432620 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:44:46.432587    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e886c2e_e044_4f32_af32_9945c531b70f.slice/crio-7d3ea754d8cda14fd4b098e0c63552dd37524919cf619504ce6759d5477517b4 WatchSource:0}: Error finding container 7d3ea754d8cda14fd4b098e0c63552dd37524919cf619504ce6759d5477517b4: Status 404 returned error can't find the container with id 7d3ea754d8cda14fd4b098e0c63552dd37524919cf619504ce6759d5477517b4
Oct 30 13:44:46.433186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.433147    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"]
Oct 30 13:44:46.435844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.435815    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:44:46.435940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.435879    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:44:46.435940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.435913    4429 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:44:46.948822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.948782    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" event={"ID":"4e886c2e-e044-4f32-af32-9945c531b70f","Type":"ContainerStarted","Data":"1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1"}
Oct 30 13:44:46.949438 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.949405    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" event={"ID":"4e886c2e-e044-4f32-af32-9945c531b70f","Type":"ContainerStarted","Data":"7d3ea754d8cda14fd4b098e0c63552dd37524919cf619504ce6759d5477517b4"}
Oct 30 13:44:46.998291 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:46.998241    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podStartSLOduration=1.9982239659999999 podStartE2EDuration="1.998223966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:46.996744945 +0000 UTC m=+4345.052238791" watchObservedRunningTime="2026-10-30 13:44:46.998223966 +0000 UTC m=+4345.053717790"
Oct 30 13:44:47.107628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:47.107585    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:47.109763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:47.109740    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xpssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 13:44:47.109869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:47.109792    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 13:44:48.107538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:48.107506    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xpssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 13:44:48.108038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:48.107566    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 13:44:49.107801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:49.107771    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xpssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 13:44:49.108210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:49.107819    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 13:44:50.107887 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:50.107858    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xpssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 13:44:50.108269 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:50.107915    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 13:44:51.108056 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:51.108027    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xpssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 13:44:51.108442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:51.108082    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 13:44:52.107767 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:52.107740    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xpssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 13:44:52.107944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:52.107797    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 13:44:53.107541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:53.107508    4429 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-xpssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 13:44:53.107945 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:53.107565    4429 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 13:44:54.764863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:54.764834    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:54.765312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:54.765252    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:44:54.766175 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:44:54.766153    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:45:00.339524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.339489    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"]
Oct 30 13:45:00.347180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.347156    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:00.369588 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.369554    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"]
Oct 30 13:45:00.390061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.389678    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"]
Oct 30 13:45:00.400070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.400049    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:00.400583 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.400551    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"]
Oct 30 13:45:00.408026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.408005    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:00.420529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.416127    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7t8\" (UniqueName: \"kubernetes.io/projected/0e22b00a-2cf6-4549-92ba-fa7019138663-kube-api-access-mj7t8\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc\" (UID: \"0e22b00a-2cf6-4549-92ba-fa7019138663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:00.420529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.416177    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fca5d-4794-489f-bb87-fedcd96d2e70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2\" (UID: \"254fca5d-4794-489f-bb87-fedcd96d2e70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:00.420529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.416295    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e22b00a-2cf6-4549-92ba-fa7019138663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc\" (UID: \"0e22b00a-2cf6-4549-92ba-fa7019138663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:00.420529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.416365    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzg8\" (UniqueName: \"kubernetes.io/projected/254fca5d-4794-489f-bb87-fedcd96d2e70-kube-api-access-nhzg8\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2\" (UID: \"254fca5d-4794-489f-bb87-fedcd96d2e70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:00.420529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.416423    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0fadc2-ca92-476c-8c21-81f946cfbe21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg\" (UID: \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:00.420529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.416541    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nqc\" (UniqueName: \"kubernetes.io/projected/1e0fadc2-ca92-476c-8c21-81f946cfbe21-kube-api-access-q2nqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg\" (UID: \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:00.421378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.421361    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"]
Oct 30 13:45:00.428795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.428777    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:00.429368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.429349    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"]
Oct 30 13:45:00.440711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.440688    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:45:00.441579 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.441561    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"]
Oct 30 13:45:00.477107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.477079    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"]
Oct 30 13:45:00.518231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.518197    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7t8\" (UniqueName: \"kubernetes.io/projected/0e22b00a-2cf6-4549-92ba-fa7019138663-kube-api-access-mj7t8\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc\" (UID: \"0e22b00a-2cf6-4549-92ba-fa7019138663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:00.518415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.518247    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fca5d-4794-489f-bb87-fedcd96d2e70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2\" (UID: \"254fca5d-4794-489f-bb87-fedcd96d2e70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:00.518415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.518311    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e22b00a-2cf6-4549-92ba-fa7019138663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc\" (UID: \"0e22b00a-2cf6-4549-92ba-fa7019138663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:00.518415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.518356    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzg8\" (UniqueName: \"kubernetes.io/projected/254fca5d-4794-489f-bb87-fedcd96d2e70-kube-api-access-nhzg8\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2\" (UID: \"254fca5d-4794-489f-bb87-fedcd96d2e70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:00.518415 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.518404    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0fadc2-ca92-476c-8c21-81f946cfbe21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg\" (UID: \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:00.518724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.518517    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nqc\" (UniqueName: \"kubernetes.io/projected/1e0fadc2-ca92-476c-8c21-81f946cfbe21-kube-api-access-q2nqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg\" (UID: \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:00.518724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.518582    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4lb\" (UniqueName: \"kubernetes.io/projected/a8bad93c-a0e2-4807-821b-08f59773174f-kube-api-access-7n4lb\") pod \"maas-api-key-cleanup-29889465-l6dgf\" (UID: \"a8bad93c-a0e2-4807-821b-08f59773174f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:00.518724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.518624    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bad93c-a0e2-4807-821b-08f59773174f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-l6dgf\" (UID: \"a8bad93c-a0e2-4807-821b-08f59773174f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:00.519042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.519019    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fca5d-4794-489f-bb87-fedcd96d2e70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2\" (UID: \"254fca5d-4794-489f-bb87-fedcd96d2e70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:00.519130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.519026    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e22b00a-2cf6-4549-92ba-fa7019138663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc\" (UID: \"0e22b00a-2cf6-4549-92ba-fa7019138663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:00.519646 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.519623    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0fadc2-ca92-476c-8c21-81f946cfbe21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg\" (UID: \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:00.543592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.543572    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7t8\" (UniqueName: \"kubernetes.io/projected/0e22b00a-2cf6-4549-92ba-fa7019138663-kube-api-access-mj7t8\") pod \"maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc\" (UID: \"0e22b00a-2cf6-4549-92ba-fa7019138663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:00.543696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.543648    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzg8\" (UniqueName: \"kubernetes.io/projected/254fca5d-4794-489f-bb87-fedcd96d2e70-kube-api-access-nhzg8\") pod \"maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2\" (UID: \"254fca5d-4794-489f-bb87-fedcd96d2e70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:00.547768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.547745    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nqc\" (UniqueName: \"kubernetes.io/projected/1e0fadc2-ca92-476c-8c21-81f946cfbe21-kube-api-access-q2nqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg\" (UID: \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:00.619783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.619720    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4lb\" (UniqueName: \"kubernetes.io/projected/a8bad93c-a0e2-4807-821b-08f59773174f-kube-api-access-7n4lb\") pod \"maas-api-key-cleanup-29889465-l6dgf\" (UID: \"a8bad93c-a0e2-4807-821b-08f59773174f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:00.619783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.619776    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bad93c-a0e2-4807-821b-08f59773174f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-l6dgf\" (UID: \"a8bad93c-a0e2-4807-821b-08f59773174f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:00.620381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.620360    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bad93c-a0e2-4807-821b-08f59773174f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-l6dgf\" (UID: \"a8bad93c-a0e2-4807-821b-08f59773174f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:00.645192 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.645168    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4lb\" (UniqueName: \"kubernetes.io/projected/a8bad93c-a0e2-4807-821b-08f59773174f-kube-api-access-7n4lb\") pod \"maas-api-key-cleanup-29889465-l6dgf\" (UID: \"a8bad93c-a0e2-4807-821b-08f59773174f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:00.691866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.691840    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:00.724936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.724908    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:00.782695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.782276    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:00.792595 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:00.792550    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:01.007358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:01.007326    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"]
Oct 30 13:45:01.013100 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:45:01.013068    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0fadc2_ca92_476c_8c21_81f946cfbe21.slice/crio-2789ec66cd609000331da02057ce116e580bd79d9c348aed97a846268e1d4bb3 WatchSource:0}: Error finding container 2789ec66cd609000331da02057ce116e580bd79d9c348aed97a846268e1d4bb3: Status 404 returned error can't find the container with id 2789ec66cd609000331da02057ce116e580bd79d9c348aed97a846268e1d4bb3
Oct 30 13:45:01.082523 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:45:01.082498    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e22b00a_2cf6_4549_92ba_fa7019138663.slice/crio-d98a2c4a34e7ae7809f693c5ab52e65747fcc2bde7703f5a30a6fb36ae161bbc WatchSource:0}: Error finding container d98a2c4a34e7ae7809f693c5ab52e65747fcc2bde7703f5a30a6fb36ae161bbc: Status 404 returned error can't find the container with id d98a2c4a34e7ae7809f693c5ab52e65747fcc2bde7703f5a30a6fb36ae161bbc
Oct 30 13:45:01.085145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:01.085111    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"]
Oct 30 13:45:01.088633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:01.088607    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" event={"ID":"1e0fadc2-ca92-476c-8c21-81f946cfbe21","Type":"ContainerStarted","Data":"2789ec66cd609000331da02057ce116e580bd79d9c348aed97a846268e1d4bb3"}
Oct 30 13:45:01.140046 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:45:01.140020    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bad93c_a0e2_4807_821b_08f59773174f.slice/crio-a32ffc5dde5bbade14bc7ff1158fd457d46ada564182dfa75007dd1d2027562b WatchSource:0}: Error finding container a32ffc5dde5bbade14bc7ff1158fd457d46ada564182dfa75007dd1d2027562b: Status 404 returned error can't find the container with id a32ffc5dde5bbade14bc7ff1158fd457d46ada564182dfa75007dd1d2027562b
Oct 30 13:45:01.140904 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:45:01.140876    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254fca5d_4794_489f_bb87_fedcd96d2e70.slice/crio-90650f01698d67b5524a18cd392d5a0eb078342aa17a686458b5a98437b2621a WatchSource:0}: Error finding container 90650f01698d67b5524a18cd392d5a0eb078342aa17a686458b5a98437b2621a: Status 404 returned error can't find the container with id 90650f01698d67b5524a18cd392d5a0eb078342aa17a686458b5a98437b2621a
Oct 30 13:45:01.141123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:01.141106    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"]
Oct 30 13:45:01.155416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:01.155383    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"]
Oct 30 13:45:01.191000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:01.190967    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"]
Oct 30 13:45:01.191218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:01.191195    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy" containerID="cri-o://3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e" gracePeriod=30
Oct 30 13:45:02.097757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.097723    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" event={"ID":"254fca5d-4794-489f-bb87-fedcd96d2e70","Type":"ContainerStarted","Data":"b9cd82a3aaa3e8e2a378e973a20006f3edb1ded1b22a95b02fecbf309e83a2b8"}
Oct 30 13:45:02.098103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.097816    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" event={"ID":"254fca5d-4794-489f-bb87-fedcd96d2e70","Type":"ContainerStarted","Data":"90650f01698d67b5524a18cd392d5a0eb078342aa17a686458b5a98437b2621a"}
Oct 30 13:45:02.099628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.099598    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" event={"ID":"a8bad93c-a0e2-4807-821b-08f59773174f","Type":"ContainerStarted","Data":"0297cb4ce14398475f18568eba05b929a9c51d5dcda7116c4cf42ba3f1f88552"}
Oct 30 13:45:02.099761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.099640    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" event={"ID":"a8bad93c-a0e2-4807-821b-08f59773174f","Type":"ContainerStarted","Data":"a32ffc5dde5bbade14bc7ff1158fd457d46ada564182dfa75007dd1d2027562b"}
Oct 30 13:45:02.102451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.102426    4429 generic.go:356] "Generic (PLEG): container finished" podID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerID="37c98bf0efd2a451efc56539437748cc63ba6acb0a615c34f63d0ef8819a6b39" exitCode=60
Oct 30 13:45:02.102554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.102499    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" event={"ID":"1e0fadc2-ca92-476c-8c21-81f946cfbe21","Type":"ContainerDied","Data":"37c98bf0efd2a451efc56539437748cc63ba6acb0a615c34f63d0ef8819a6b39"}
Oct 30 13:45:02.102719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.102695    4429 scope.go:122] "RemoveContainer" containerID="37c98bf0efd2a451efc56539437748cc63ba6acb0a615c34f63d0ef8819a6b39"
Oct 30 13:45:02.104325 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.104256    4429 generic.go:356] "Generic (PLEG): container finished" podID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerID="8ebaebdbf3ddb3d1e3cc724fb141af4b0c2cbfe7c06de9ba5857e1d7450a2f6d" exitCode=60
Oct 30 13:45:02.104450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.104348    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" event={"ID":"0e22b00a-2cf6-4549-92ba-fa7019138663","Type":"ContainerDied","Data":"8ebaebdbf3ddb3d1e3cc724fb141af4b0c2cbfe7c06de9ba5857e1d7450a2f6d"}
Oct 30 13:45:02.104450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.104386    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" event={"ID":"0e22b00a-2cf6-4549-92ba-fa7019138663","Type":"ContainerStarted","Data":"d98a2c4a34e7ae7809f693c5ab52e65747fcc2bde7703f5a30a6fb36ae161bbc"}
Oct 30 13:45:02.104612 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.104597    4429 scope.go:122] "RemoveContainer" containerID="8ebaebdbf3ddb3d1e3cc724fb141af4b0c2cbfe7c06de9ba5857e1d7450a2f6d"
Oct 30 13:45:02.147411 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.147372    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" podStartSLOduration=2.147358809 podStartE2EDuration="2.147358809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.141095771 +0000 UTC m=+4360.196589594" watchObservedRunningTime="2026-10-30 13:45:02.147358809 +0000 UTC m=+4360.202852632"
Oct 30 13:45:02.232549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:02.232500    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" podStartSLOduration=2.23248479 podStartE2EDuration="2.23248479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.227477851 +0000 UTC m=+4360.282971677" watchObservedRunningTime="2026-10-30 13:45:02.23248479 +0000 UTC m=+4360.287978613"
Oct 30 13:45:03.113750 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.113720    4429 generic.go:356] "Generic (PLEG): container finished" podID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerID="b9cd82a3aaa3e8e2a378e973a20006f3edb1ded1b22a95b02fecbf309e83a2b8" exitCode=60
Oct 30 13:45:03.114216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.113807    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" event={"ID":"254fca5d-4794-489f-bb87-fedcd96d2e70","Type":"ContainerDied","Data":"b9cd82a3aaa3e8e2a378e973a20006f3edb1ded1b22a95b02fecbf309e83a2b8"}
Oct 30 13:45:03.114216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.114093    4429 scope.go:122] "RemoveContainer" containerID="b9cd82a3aaa3e8e2a378e973a20006f3edb1ded1b22a95b02fecbf309e83a2b8"
Oct 30 13:45:03.115689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.115669    4429 generic.go:356] "Generic (PLEG): container finished" podID="a8bad93c-a0e2-4807-821b-08f59773174f" containerID="0297cb4ce14398475f18568eba05b929a9c51d5dcda7116c4cf42ba3f1f88552" exitCode=60
Oct 30 13:45:03.115815 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.115702    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" event={"ID":"a8bad93c-a0e2-4807-821b-08f59773174f","Type":"ContainerDied","Data":"0297cb4ce14398475f18568eba05b929a9c51d5dcda7116c4cf42ba3f1f88552"}
Oct 30 13:45:03.116094 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.115997    4429 scope.go:122] "RemoveContainer" containerID="0297cb4ce14398475f18568eba05b929a9c51d5dcda7116c4cf42ba3f1f88552"
Oct 30 13:45:03.118356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.118270    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" event={"ID":"1e0fadc2-ca92-476c-8c21-81f946cfbe21","Type":"ContainerStarted","Data":"0dabad0b3898b36106b3a0a66d6f718c4955a3ed4bf0acfc267504525e408a32"}
Oct 30 13:45:03.120702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.120680    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" event={"ID":"0e22b00a-2cf6-4549-92ba-fa7019138663","Type":"ContainerStarted","Data":"4a826b98ab0d1ddb690dc22c022a6462afb8f14000efc62896ded9311f8468f6"}
Oct 30 13:45:03.256210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.256154    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" podStartSLOduration=3.256139637 podStartE2EDuration="3.256139637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.252639069 +0000 UTC m=+4361.308132893" watchObservedRunningTime="2026-10-30 13:45:03.256139637 +0000 UTC m=+4361.311633459"
Oct 30 13:45:03.300833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:03.300783    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" podStartSLOduration=3.300769187 podStartE2EDuration="3.300769187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.297146599 +0000 UTC m=+4361.352640421" watchObservedRunningTime="2026-10-30 13:45:03.300769187 +0000 UTC m=+4361.356263010"
Oct 30 13:45:04.130352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.130318    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" event={"ID":"a8bad93c-a0e2-4807-821b-08f59773174f","Type":"ContainerStarted","Data":"25308f73695caf8446769ae61ccd83f5f3a86d7b170c4643020a477b46a0b843"}
Oct 30 13:45:04.132210 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.132184    4429 generic.go:356] "Generic (PLEG): container finished" podID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerID="0dabad0b3898b36106b3a0a66d6f718c4955a3ed4bf0acfc267504525e408a32" exitCode=60
Oct 30 13:45:04.132340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.132264    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" event={"ID":"1e0fadc2-ca92-476c-8c21-81f946cfbe21","Type":"ContainerDied","Data":"0dabad0b3898b36106b3a0a66d6f718c4955a3ed4bf0acfc267504525e408a32"}
Oct 30 13:45:04.132340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.132324    4429 scope.go:122] "RemoveContainer" containerID="37c98bf0efd2a451efc56539437748cc63ba6acb0a615c34f63d0ef8819a6b39"
Oct 30 13:45:04.132541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.132528    4429 scope.go:122] "RemoveContainer" containerID="0dabad0b3898b36106b3a0a66d6f718c4955a3ed4bf0acfc267504525e408a32"
Oct 30 13:45:04.132913 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:04.132886    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg_odh-ai-gateway-infra(1e0fadc2-ca92-476c-8c21-81f946cfbe21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21"
Oct 30 13:45:04.134420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.134402    4429 generic.go:356] "Generic (PLEG): container finished" podID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerID="4a826b98ab0d1ddb690dc22c022a6462afb8f14000efc62896ded9311f8468f6" exitCode=60
Oct 30 13:45:04.134512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.134469    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" event={"ID":"0e22b00a-2cf6-4549-92ba-fa7019138663","Type":"ContainerDied","Data":"4a826b98ab0d1ddb690dc22c022a6462afb8f14000efc62896ded9311f8468f6"}
Oct 30 13:45:04.134786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.134767    4429 scope.go:122] "RemoveContainer" containerID="4a826b98ab0d1ddb690dc22c022a6462afb8f14000efc62896ded9311f8468f6"
Oct 30 13:45:04.135124 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:04.135105    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc_odh-ai-gateway-infra(0e22b00a-2cf6-4549-92ba-fa7019138663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663"
Oct 30 13:45:04.136385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.136368    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" event={"ID":"254fca5d-4794-489f-bb87-fedcd96d2e70","Type":"ContainerStarted","Data":"bf96f31f681c924b8155a9a4644dae55aca83a214d376014c10add9e33ca8afc"}
Oct 30 13:45:04.146619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:04.146597    4429 scope.go:122] "RemoveContainer" containerID="8ebaebdbf3ddb3d1e3cc724fb141af4b0c2cbfe7c06de9ba5857e1d7450a2f6d"
Oct 30 13:45:05.147254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.147225    4429 scope.go:122] "RemoveContainer" containerID="4a826b98ab0d1ddb690dc22c022a6462afb8f14000efc62896ded9311f8468f6"
Oct 30 13:45:05.147804 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:05.147781    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc_odh-ai-gateway-infra(0e22b00a-2cf6-4549-92ba-fa7019138663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663"
Oct 30 13:45:05.149271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.149240    4429 generic.go:356] "Generic (PLEG): container finished" podID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerID="bf96f31f681c924b8155a9a4644dae55aca83a214d376014c10add9e33ca8afc" exitCode=60
Oct 30 13:45:05.149427 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.149336    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" event={"ID":"254fca5d-4794-489f-bb87-fedcd96d2e70","Type":"ContainerDied","Data":"bf96f31f681c924b8155a9a4644dae55aca83a214d376014c10add9e33ca8afc"}
Oct 30 13:45:05.149500 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.149463    4429 scope.go:122] "RemoveContainer" containerID="b9cd82a3aaa3e8e2a378e973a20006f3edb1ded1b22a95b02fecbf309e83a2b8"
Oct 30 13:45:05.149818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.149796    4429 scope.go:122] "RemoveContainer" containerID="bf96f31f681c924b8155a9a4644dae55aca83a214d376014c10add9e33ca8afc"
Oct 30 13:45:05.150199 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:05.150179    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2_odh-ai-gateway-infra(254fca5d-4794-489f-bb87-fedcd96d2e70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70"
Oct 30 13:45:05.151641 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.151619    4429 generic.go:356] "Generic (PLEG): container finished" podID="a8bad93c-a0e2-4807-821b-08f59773174f" containerID="25308f73695caf8446769ae61ccd83f5f3a86d7b170c4643020a477b46a0b843" exitCode=60
Oct 30 13:45:05.151737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.151698    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" event={"ID":"a8bad93c-a0e2-4807-821b-08f59773174f","Type":"ContainerDied","Data":"25308f73695caf8446769ae61ccd83f5f3a86d7b170c4643020a477b46a0b843"}
Oct 30 13:45:05.151972 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.151956    4429 scope.go:122] "RemoveContainer" containerID="25308f73695caf8446769ae61ccd83f5f3a86d7b170c4643020a477b46a0b843"
Oct 30 13:45:05.152344 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:05.152324    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-l6dgf_odh-ai-gateway-infra(a8bad93c-a0e2-4807-821b-08f59773174f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" podUID="a8bad93c-a0e2-4807-821b-08f59773174f"
Oct 30 13:45:05.153741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.153721    4429 scope.go:122] "RemoveContainer" containerID="0dabad0b3898b36106b3a0a66d6f718c4955a3ed4bf0acfc267504525e408a32"
Oct 30 13:45:05.153989 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:05.153969    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg_odh-ai-gateway-infra(1e0fadc2-ca92-476c-8c21-81f946cfbe21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21"
Oct 30 13:45:05.164378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:05.164349    4429 scope.go:122] "RemoveContainer" containerID="0297cb4ce14398475f18568eba05b929a9c51d5dcda7116c4cf42ba3f1f88552"
Oct 30 13:45:06.164161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.164133    4429 scope.go:122] "RemoveContainer" containerID="bf96f31f681c924b8155a9a4644dae55aca83a214d376014c10add9e33ca8afc"
Oct 30 13:45:06.164630 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:06.164608    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2_odh-ai-gateway-infra(254fca5d-4794-489f-bb87-fedcd96d2e70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70"
Oct 30 13:45:06.165750 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.165727    4429 scope.go:122] "RemoveContainer" containerID="25308f73695caf8446769ae61ccd83f5f3a86d7b170c4643020a477b46a0b843"
Oct 30 13:45:06.166089 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:06.166068    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-l6dgf_odh-ai-gateway-infra(a8bad93c-a0e2-4807-821b-08f59773174f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" podUID="a8bad93c-a0e2-4807-821b-08f59773174f"
Oct 30 13:45:06.491790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.491763    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:45:06.631407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631374    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-kube-api-access-sxr7r\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-kube-api-access-sxr7r\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631432    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-data\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631549    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-credential-socket\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631570    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-token\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-token\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631632    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istiod-ca-cert\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631694    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-podinfo\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631713    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-socket\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631758    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-certs\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631778    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-data" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:06.631964 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631952    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-envoy\") pod \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\" (UID: \"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32\") "
Oct 30 13:45:06.631998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.631974    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-certs" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:06.632112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.632091    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istiod-ca-cert" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:06.632439 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.632414    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-credential-socket" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:06.632641 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.632566    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-socket" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:06.633011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.632987    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:06.633140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.633019    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:06.633140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.633034    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:06.633140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.633048    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:06.633140 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.633062    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:06.634184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.634159    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-kube-api-access-sxr7r" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "kube-api-access-sxr7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:06.634448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.634419    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-podinfo" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:06.634536 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.634473    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-token" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:06.634818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.634801    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-envoy" pod "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" (UID: "b84945ac-ece7-47e8-8bd7-3eb6bbba1c32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:06.735186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.735110    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxr7r\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-kube-api-access-sxr7r\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:06.735186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.735136    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:06.735186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.735145    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:06.735186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:06.735155    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:07.176336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.176244    4429 generic.go:356] "Generic (PLEG): container finished" podID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerID="3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e" exitCode=0
Oct 30 13:45:07.176336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.176322    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"
Oct 30 13:45:07.176336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.176321    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" event={"ID":"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32","Type":"ContainerDied","Data":"3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e"}
Oct 30 13:45:07.176909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.176387    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5" event={"ID":"b84945ac-ece7-47e8-8bd7-3eb6bbba1c32","Type":"ContainerDied","Data":"3798c3d4d972543dbecea2fe032e33c258af1fbd681bc391f8dc47c719797563"}
Oct 30 13:45:07.176909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.176417    4429 scope.go:122] "RemoveContainer" containerID="3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e"
Oct 30 13:45:07.191518 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.191483    4429 scope.go:122] "RemoveContainer" containerID="3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e"
Oct 30 13:45:07.192317 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:07.192272    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e\": container with ID starting with 3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e not found: ID does not exist" containerID="3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e"
Oct 30 13:45:07.192428 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.192319    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e"} err="failed to get container status \"3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e\": rpc error: code = NotFound desc = could not find container \"3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e\": container with ID starting with 3624997d499e3a6d4b34f1bc9fd823dbb9b4763bb4e185495301b2cd26a5995e not found: ID does not exist"
Oct 30 13:45:07.252058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.252027    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"]
Oct 30 13:45:07.268367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.268334    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-cvrs5"]
Oct 30 13:45:07.522144 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.522117    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"]
Oct 30 13:45:07.523049 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.523034    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy"
Oct 30 13:45:07.523104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.523051    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy"
Oct 30 13:45:07.523178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.523169    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" containerName="istio-proxy"
Oct 30 13:45:07.528313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.528275    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.563844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.563819    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:45:07.569135 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.569108    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"]
Oct 30 13:45:07.651630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.651572    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvrk\" (UniqueName: \"kubernetes.io/projected/641780cb-2978-4633-af57-2463feb3acef-kube-api-access-qbvrk\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.651945 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.651921    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641780cb-2978-4633-af57-2463feb3acef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.652277 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.652251    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641780cb-2978-4633-af57-2463feb3acef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.760308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.754436    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvrk\" (UniqueName: \"kubernetes.io/projected/641780cb-2978-4633-af57-2463feb3acef-kube-api-access-qbvrk\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.760308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.754721    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641780cb-2978-4633-af57-2463feb3acef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.760308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.754826    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641780cb-2978-4633-af57-2463feb3acef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.760308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.758558    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641780cb-2978-4633-af57-2463feb3acef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.763679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.763650    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641780cb-2978-4633-af57-2463feb3acef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.791046 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.790979    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvrk\" (UniqueName: \"kubernetes.io/projected/641780cb-2978-4633-af57-2463feb3acef-kube-api-access-qbvrk\") pod \"maas-api-revoke-sub-a0855dd6-9r8r7\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:07.843008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:07.842982    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:08.078045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:08.077966    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84945ac-ece7-47e8-8bd7-3eb6bbba1c32" path="/var/lib/kubelet/pods/b84945ac-ece7-47e8-8bd7-3eb6bbba1c32/volumes"
Oct 30 13:45:08.183957 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:45:08.183918    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641780cb_2978_4633_af57_2463feb3acef.slice/crio-fb9f283bb91f6f9a17c6dbad68956bb8632ee1d2bc297c5955544f210ba90752 WatchSource:0}: Error finding container fb9f283bb91f6f9a17c6dbad68956bb8632ee1d2bc297c5955544f210ba90752: Status 404 returned error can't find the container with id fb9f283bb91f6f9a17c6dbad68956bb8632ee1d2bc297c5955544f210ba90752
Oct 30 13:45:08.187330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:08.186803    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"]
Oct 30 13:45:09.213399 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:09.213363    4429 generic.go:356] "Generic (PLEG): container finished" podID="641780cb-2978-4633-af57-2463feb3acef" containerID="d565912c0d6814fd1955eac172b9a28bf8eb94f239811caf22041038d788c5ed" exitCode=0
Oct 30 13:45:09.213782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:09.213437    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7" event={"ID":"641780cb-2978-4633-af57-2463feb3acef","Type":"ContainerDied","Data":"d565912c0d6814fd1955eac172b9a28bf8eb94f239811caf22041038d788c5ed"}
Oct 30 13:45:09.213782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:09.213483    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7" event={"ID":"641780cb-2978-4633-af57-2463feb3acef","Type":"ContainerStarted","Data":"fb9f283bb91f6f9a17c6dbad68956bb8632ee1d2bc297c5955544f210ba90752"}
Oct 30 13:45:10.400868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.400846    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:10.498522 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.498440    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641780cb-2978-4633-af57-2463feb3acef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641780cb-2978-4633-af57-2463feb3acef-maas-api-openshift-ca\") pod \"641780cb-2978-4633-af57-2463feb3acef\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") "
Oct 30 13:45:10.498717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.498590    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641780cb-2978-4633-af57-2463feb3acef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641780cb-2978-4633-af57-2463feb3acef-maas-api-xks-ca\") pod \"641780cb-2978-4633-af57-2463feb3acef\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") "
Oct 30 13:45:10.498717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.498695    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641780cb-2978-4633-af57-2463feb3acef-kube-api-access-qbvrk\" (UniqueName: \"kubernetes.io/projected/641780cb-2978-4633-af57-2463feb3acef-kube-api-access-qbvrk\") pod \"641780cb-2978-4633-af57-2463feb3acef\" (UID: \"641780cb-2978-4633-af57-2463feb3acef\") "
Oct 30 13:45:10.498895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.498867    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641780cb-2978-4633-af57-2463feb3acef-maas-api-openshift-ca" pod "641780cb-2978-4633-af57-2463feb3acef" (UID: "641780cb-2978-4633-af57-2463feb3acef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:10.499463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.499442    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641780cb-2978-4633-af57-2463feb3acef-maas-api-openshift-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:10.501529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.501506    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641780cb-2978-4633-af57-2463feb3acef-kube-api-access-qbvrk" pod "641780cb-2978-4633-af57-2463feb3acef" (UID: "641780cb-2978-4633-af57-2463feb3acef"). InnerVolumeSpecName "kube-api-access-qbvrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:10.501529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.501511    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641780cb-2978-4633-af57-2463feb3acef-maas-api-xks-ca" pod "641780cb-2978-4633-af57-2463feb3acef" (UID: "641780cb-2978-4633-af57-2463feb3acef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:10.600876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.600845    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641780cb-2978-4633-af57-2463feb3acef-maas-api-xks-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:10.600876 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:10.600876    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvrk\" (UniqueName: \"kubernetes.io/projected/641780cb-2978-4633-af57-2463feb3acef-kube-api-access-qbvrk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:11.231130 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:11.231103    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"
Oct 30 13:45:11.231339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:11.231179    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7" event={"ID":"641780cb-2978-4633-af57-2463feb3acef","Type":"ContainerDied","Data":"fb9f283bb91f6f9a17c6dbad68956bb8632ee1d2bc297c5955544f210ba90752"}
Oct 30 13:45:11.231339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:11.231234    4429 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb9f283bb91f6f9a17c6dbad68956bb8632ee1d2bc297c5955544f210ba90752"
Oct 30 13:45:12.372843 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:12.372808    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:45:14.077860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:14.077833    4429 scope.go:122] "RemoveContainer" containerID="4a826b98ab0d1ddb690dc22c022a6462afb8f14000efc62896ded9311f8468f6"
Oct 30 13:45:14.861918 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:14.861878    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"]
Oct 30 13:45:14.862199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:14.862172    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z" podUID="c241fb9d-2a90-474e-be70-2abca4252e27" containerName="maas-api" containerID="cri-o://645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c" gracePeriod=30
Oct 30 13:45:15.207503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.207473    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:45:15.303345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.302210    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-maas-api-tls\") pod \"c241fb9d-2a90-474e-be70-2abca4252e27\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") "
Oct 30 13:45:15.303345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.302270    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-metrics-certs\") pod \"c241fb9d-2a90-474e-be70-2abca4252e27\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") "
Oct 30 13:45:15.303345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.302356    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c241fb9d-2a90-474e-be70-2abca4252e27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c241fb9d-2a90-474e-be70-2abca4252e27-openshift-service-ca\") pod \"c241fb9d-2a90-474e-be70-2abca4252e27\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") "
Oct 30 13:45:15.303345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.302417    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c241fb9d-2a90-474e-be70-2abca4252e27-kube-api-access-7w4sq\" (UniqueName: \"kubernetes.io/projected/c241fb9d-2a90-474e-be70-2abca4252e27-kube-api-access-7w4sq\") pod \"c241fb9d-2a90-474e-be70-2abca4252e27\" (UID: \"c241fb9d-2a90-474e-be70-2abca4252e27\") "
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.306040    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c241fb9d-2a90-474e-be70-2abca4252e27-kube-api-access-7w4sq" pod "c241fb9d-2a90-474e-be70-2abca4252e27" (UID: "c241fb9d-2a90-474e-be70-2abca4252e27"). InnerVolumeSpecName "kube-api-access-7w4sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.306776    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c241fb9d-2a90-474e-be70-2abca4252e27-openshift-service-ca" pod "c241fb9d-2a90-474e-be70-2abca4252e27" (UID: "c241fb9d-2a90-474e-be70-2abca4252e27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.308243    4429 generic.go:356] "Generic (PLEG): container finished" podID="c241fb9d-2a90-474e-be70-2abca4252e27" containerID="645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c" exitCode=0
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.308436    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z" event={"ID":"c241fb9d-2a90-474e-be70-2abca4252e27","Type":"ContainerDied","Data":"645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c"}
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.308490    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z" event={"ID":"c241fb9d-2a90-474e-be70-2abca4252e27","Type":"ContainerDied","Data":"9957ef82084f6a1ee38986d08242f5254d554e1d3ca5c3cb53a00c47b4e8b6c7"}
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.308587    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.310188    4429 scope.go:122] "RemoveContainer" containerID="645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c"
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.313603    4429 generic.go:356] "Generic (PLEG): container finished" podID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerID="bbcc6667c38e902e487637192f10dd44d83999216730af430b71e4d20de278df" exitCode=60
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.313806    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" event={"ID":"0e22b00a-2cf6-4549-92ba-fa7019138663","Type":"ContainerDied","Data":"bbcc6667c38e902e487637192f10dd44d83999216730af430b71e4d20de278df"}
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.314328    4429 scope.go:122] "RemoveContainer" containerID="bbcc6667c38e902e487637192f10dd44d83999216730af430b71e4d20de278df"
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:15.314728    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc_odh-ai-gateway-infra(0e22b00a-2cf6-4549-92ba-fa7019138663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663"
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.315429    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-metrics-certs" pod "c241fb9d-2a90-474e-be70-2abca4252e27" (UID: "c241fb9d-2a90-474e-be70-2abca4252e27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:15.321343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.316410    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-maas-api-tls" pod "c241fb9d-2a90-474e-be70-2abca4252e27" (UID: "c241fb9d-2a90-474e-be70-2abca4252e27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:15.355156 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.355124    4429 scope.go:122] "RemoveContainer" containerID="645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c"
Oct 30 13:45:15.355660 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:15.355633    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c\": container with ID starting with 645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c not found: ID does not exist" containerID="645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c"
Oct 30 13:45:15.355797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.355691    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c"} err="failed to get container status \"645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c\": rpc error: code = NotFound desc = could not find container \"645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c\": container with ID starting with 645fd41e65e3f65ee8f5072143ea69ef47650574a1f57c83b4b4c8644ba5686c not found: ID does not exist"
Oct 30 13:45:15.355797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.355716    4429 scope.go:122] "RemoveContainer" containerID="4a826b98ab0d1ddb690dc22c022a6462afb8f14000efc62896ded9311f8468f6"
Oct 30 13:45:15.404990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.404961    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:15.404990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.404987    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c241fb9d-2a90-474e-be70-2abca4252e27-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:15.404990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.404996    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c241fb9d-2a90-474e-be70-2abca4252e27-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:15.405233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.405008    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w4sq\" (UniqueName: \"kubernetes.io/projected/c241fb9d-2a90-474e-be70-2abca4252e27-kube-api-access-7w4sq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:15.663074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.662979    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"]
Oct 30 13:45:15.672067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:15.672032    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-22efcf-56d98df895-t5d7z"]
Oct 30 13:45:16.081234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:16.081200    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c241fb9d-2a90-474e-be70-2abca4252e27" path="/var/lib/kubelet/pods/c241fb9d-2a90-474e-be70-2abca4252e27/volumes"
Oct 30 13:45:16.390702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:16.390604    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"]
Oct 30 13:45:16.915317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:16.915258    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:16.944315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:16.940505    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e22b00a-2cf6-4549-92ba-fa7019138663-kube-api-access-mj7t8\" (UniqueName: \"kubernetes.io/projected/0e22b00a-2cf6-4549-92ba-fa7019138663-kube-api-access-mj7t8\") pod \"0e22b00a-2cf6-4549-92ba-fa7019138663\" (UID: \"0e22b00a-2cf6-4549-92ba-fa7019138663\") "
Oct 30 13:45:16.944315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:16.940570    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e22b00a-2cf6-4549-92ba-fa7019138663-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e22b00a-2cf6-4549-92ba-fa7019138663-maas-api-service-ca\") pod \"0e22b00a-2cf6-4549-92ba-fa7019138663\" (UID: \"0e22b00a-2cf6-4549-92ba-fa7019138663\") "
Oct 30 13:45:16.986684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:16.986651    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e22b00a-2cf6-4549-92ba-fa7019138663-kube-api-access-mj7t8" pod "0e22b00a-2cf6-4549-92ba-fa7019138663" (UID: "0e22b00a-2cf6-4549-92ba-fa7019138663"). InnerVolumeSpecName "kube-api-access-mj7t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:17.014795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.012905    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e22b00a-2cf6-4549-92ba-fa7019138663-maas-api-service-ca" pod "0e22b00a-2cf6-4549-92ba-fa7019138663" (UID: "0e22b00a-2cf6-4549-92ba-fa7019138663"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:17.052414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.046554    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj7t8\" (UniqueName: \"kubernetes.io/projected/0e22b00a-2cf6-4549-92ba-fa7019138663-kube-api-access-mj7t8\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:17.052414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.046588    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e22b00a-2cf6-4549-92ba-fa7019138663-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:17.069909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.069858    4429 scope.go:122] "RemoveContainer" containerID="bf96f31f681c924b8155a9a4644dae55aca83a214d376014c10add9e33ca8afc"
Oct 30 13:45:17.335859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.335782    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"
Oct 30 13:45:17.335859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.335790    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc" event={"ID":"0e22b00a-2cf6-4549-92ba-fa7019138663","Type":"ContainerDied","Data":"d98a2c4a34e7ae7809f693c5ab52e65747fcc2bde7703f5a30a6fb36ae161bbc"}
Oct 30 13:45:17.336085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.335866    4429 scope.go:122] "RemoveContainer" containerID="bbcc6667c38e902e487637192f10dd44d83999216730af430b71e4d20de278df"
Oct 30 13:45:17.425454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.425377    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"]
Oct 30 13:45:17.467222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:17.463795    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-75d385-29889465-gtnhc"]
Oct 30 13:45:18.076365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:18.075831    4429 scope.go:122] "RemoveContainer" containerID="0dabad0b3898b36106b3a0a66d6f718c4955a3ed4bf0acfc267504525e408a32"
Oct 30 13:45:18.172330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:18.155205    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" path="/var/lib/kubelet/pods/0e22b00a-2cf6-4549-92ba-fa7019138663/volumes"
Oct 30 13:45:18.347051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:18.346953    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" event={"ID":"254fca5d-4794-489f-bb87-fedcd96d2e70","Type":"ContainerStarted","Data":"f621a5d03a6502fbcd6f0add81aa3191311fe8dd814eba41b2eb0fa1a7698c46"}
Oct 30 13:45:19.094312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.074688    4429 scope.go:122] "RemoveContainer" containerID="25308f73695caf8446769ae61ccd83f5f3a86d7b170c4643020a477b46a0b843"
Oct 30 13:45:19.435253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.435222    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"]
Oct 30 13:45:19.439892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.439852    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" event={"ID":"1e0fadc2-ca92-476c-8c21-81f946cfbe21","Type":"ContainerStarted","Data":"42d2b9d2933d2441bddd6826780f223ae6c891a56fa4415dcfba21e220d9499e"}
Oct 30 13:45:19.442263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.442237    4429 generic.go:356] "Generic (PLEG): container finished" podID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerID="f621a5d03a6502fbcd6f0add81aa3191311fe8dd814eba41b2eb0fa1a7698c46" exitCode=60
Oct 30 13:45:19.442393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.442333    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" event={"ID":"254fca5d-4794-489f-bb87-fedcd96d2e70","Type":"ContainerDied","Data":"f621a5d03a6502fbcd6f0add81aa3191311fe8dd814eba41b2eb0fa1a7698c46"}
Oct 30 13:45:19.442393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.442376    4429 scope.go:122] "RemoveContainer" containerID="bf96f31f681c924b8155a9a4644dae55aca83a214d376014c10add9e33ca8afc"
Oct 30 13:45:19.771818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.771791    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:19.776504 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.776480    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254fca5d-4794-489f-bb87-fedcd96d2e70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fca5d-4794-489f-bb87-fedcd96d2e70-maas-api-service-ca\") pod \"254fca5d-4794-489f-bb87-fedcd96d2e70\" (UID: \"254fca5d-4794-489f-bb87-fedcd96d2e70\") "
Oct 30 13:45:19.776766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.776749    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254fca5d-4794-489f-bb87-fedcd96d2e70-kube-api-access-nhzg8\" (UniqueName: \"kubernetes.io/projected/254fca5d-4794-489f-bb87-fedcd96d2e70-kube-api-access-nhzg8\") pod \"254fca5d-4794-489f-bb87-fedcd96d2e70\" (UID: \"254fca5d-4794-489f-bb87-fedcd96d2e70\") "
Oct 30 13:45:19.776846 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.776785    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254fca5d-4794-489f-bb87-fedcd96d2e70-maas-api-service-ca" pod "254fca5d-4794-489f-bb87-fedcd96d2e70" (UID: "254fca5d-4794-489f-bb87-fedcd96d2e70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:19.777440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.777382    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fca5d-4794-489f-bb87-fedcd96d2e70-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.779467 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.779442    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254fca5d-4794-489f-bb87-fedcd96d2e70-kube-api-access-nhzg8" pod "254fca5d-4794-489f-bb87-fedcd96d2e70" (UID: "254fca5d-4794-489f-bb87-fedcd96d2e70"). InnerVolumeSpecName "kube-api-access-nhzg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:19.883005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:19.882964    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhzg8\" (UniqueName: \"kubernetes.io/projected/254fca5d-4794-489f-bb87-fedcd96d2e70-kube-api-access-nhzg8\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.452658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.452624    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2" event={"ID":"254fca5d-4794-489f-bb87-fedcd96d2e70","Type":"ContainerDied","Data":"90650f01698d67b5524a18cd392d5a0eb078342aa17a686458b5a98437b2621a"}
Oct 30 13:45:20.452658 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.452649    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"
Oct 30 13:45:20.453511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.452697    4429 scope.go:122] "RemoveContainer" containerID="f621a5d03a6502fbcd6f0add81aa3191311fe8dd814eba41b2eb0fa1a7698c46"
Oct 30 13:45:20.455753 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.455720    4429 generic.go:356] "Generic (PLEG): container finished" podID="a8bad93c-a0e2-4807-821b-08f59773174f" containerID="d1c2e20756bfa08bce865152a47f64ba2b87fd9fd576c7671ebeb79634df2893" exitCode=60
Oct 30 13:45:20.455915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.455899    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" event={"ID":"a8bad93c-a0e2-4807-821b-08f59773174f","Type":"ContainerDied","Data":"d1c2e20756bfa08bce865152a47f64ba2b87fd9fd576c7671ebeb79634df2893"}
Oct 30 13:45:20.456379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.456355    4429 scope.go:122] "RemoveContainer" containerID="d1c2e20756bfa08bce865152a47f64ba2b87fd9fd576c7671ebeb79634df2893"
Oct 30 13:45:20.456757 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:20.456733    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-l6dgf_odh-ai-gateway-infra(a8bad93c-a0e2-4807-821b-08f59773174f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" podUID="a8bad93c-a0e2-4807-821b-08f59773174f"
Oct 30 13:45:20.458782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.458759    4429 generic.go:356] "Generic (PLEG): container finished" podID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerID="42d2b9d2933d2441bddd6826780f223ae6c891a56fa4415dcfba21e220d9499e" exitCode=60
Oct 30 13:45:20.458877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.458823    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" event={"ID":"1e0fadc2-ca92-476c-8c21-81f946cfbe21","Type":"ContainerDied","Data":"42d2b9d2933d2441bddd6826780f223ae6c891a56fa4415dcfba21e220d9499e"}
Oct 30 13:45:20.459226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.459210    4429 scope.go:122] "RemoveContainer" containerID="42d2b9d2933d2441bddd6826780f223ae6c891a56fa4415dcfba21e220d9499e"
Oct 30 13:45:20.459573 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:20.459550    4429 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg_odh-ai-gateway-infra(1e0fadc2-ca92-476c-8c21-81f946cfbe21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21"
Oct 30 13:45:20.465322 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.465297    4429 scope.go:122] "RemoveContainer" containerID="25308f73695caf8446769ae61ccd83f5f3a86d7b170c4643020a477b46a0b843"
Oct 30 13:45:20.480480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.480458    4429 scope.go:122] "RemoveContainer" containerID="0dabad0b3898b36106b3a0a66d6f718c4955a3ed4bf0acfc267504525e408a32"
Oct 30 13:45:20.516462 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.516418    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"]
Oct 30 13:45:20.613028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.612419    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"]
Oct 30 13:45:20.623597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:20.623551    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6ff0e6-29889465-mqcv2"]
Oct 30 13:45:21.544639 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.544597    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"]
Oct 30 13:45:21.726320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.725938    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:21.791330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.788059    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:21.814371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.814338    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0fadc2-ca92-476c-8c21-81f946cfbe21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0fadc2-ca92-476c-8c21-81f946cfbe21-maas-api-service-ca\") pod \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\" (UID: \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\") "
Oct 30 13:45:21.814554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.814490    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0fadc2-ca92-476c-8c21-81f946cfbe21-kube-api-access-q2nqc\" (UniqueName: \"kubernetes.io/projected/1e0fadc2-ca92-476c-8c21-81f946cfbe21-kube-api-access-q2nqc\") pod \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\" (UID: \"1e0fadc2-ca92-476c-8c21-81f946cfbe21\") "
Oct 30 13:45:21.814661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.814643    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bad93c-a0e2-4807-821b-08f59773174f-kube-api-access-7n4lb\" (UniqueName: \"kubernetes.io/projected/a8bad93c-a0e2-4807-821b-08f59773174f-kube-api-access-7n4lb\") pod \"a8bad93c-a0e2-4807-821b-08f59773174f\" (UID: \"a8bad93c-a0e2-4807-821b-08f59773174f\") "
Oct 30 13:45:21.814751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.814682    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bad93c-a0e2-4807-821b-08f59773174f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bad93c-a0e2-4807-821b-08f59773174f-maas-api-service-ca\") pod \"a8bad93c-a0e2-4807-821b-08f59773174f\" (UID: \"a8bad93c-a0e2-4807-821b-08f59773174f\") "
Oct 30 13:45:21.814751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.814686    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0fadc2-ca92-476c-8c21-81f946cfbe21-maas-api-service-ca" pod "1e0fadc2-ca92-476c-8c21-81f946cfbe21" (UID: "1e0fadc2-ca92-476c-8c21-81f946cfbe21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:21.815113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.815084    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bad93c-a0e2-4807-821b-08f59773174f-maas-api-service-ca" pod "a8bad93c-a0e2-4807-821b-08f59773174f" (UID: "a8bad93c-a0e2-4807-821b-08f59773174f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:21.815412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.815393    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bad93c-a0e2-4807-821b-08f59773174f-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.815464 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.815421    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0fadc2-ca92-476c-8c21-81f946cfbe21-maas-api-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.817059 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.817020    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0fadc2-ca92-476c-8c21-81f946cfbe21-kube-api-access-q2nqc" pod "1e0fadc2-ca92-476c-8c21-81f946cfbe21" (UID: "1e0fadc2-ca92-476c-8c21-81f946cfbe21"). InnerVolumeSpecName "kube-api-access-q2nqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:21.817563 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.817543    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bad93c-a0e2-4807-821b-08f59773174f-kube-api-access-7n4lb" pod "a8bad93c-a0e2-4807-821b-08f59773174f" (UID: "a8bad93c-a0e2-4807-821b-08f59773174f"). InnerVolumeSpecName "kube-api-access-7n4lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:21.917430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.917379    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nqc\" (UniqueName: \"kubernetes.io/projected/1e0fadc2-ca92-476c-8c21-81f946cfbe21-kube-api-access-q2nqc\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.917680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:21.917665    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n4lb\" (UniqueName: \"kubernetes.io/projected/a8bad93c-a0e2-4807-821b-08f59773174f-kube-api-access-7n4lb\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:22.082586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.082497    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" path="/var/lib/kubelet/pods/254fca5d-4794-489f-bb87-fedcd96d2e70/volumes"
Oct 30 13:45:22.482356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.482321    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf" event={"ID":"a8bad93c-a0e2-4807-821b-08f59773174f","Type":"ContainerDied","Data":"a32ffc5dde5bbade14bc7ff1158fd457d46ada564182dfa75007dd1d2027562b"}
Oct 30 13:45:22.482555 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.482399    4429 scope.go:122] "RemoveContainer" containerID="d1c2e20756bfa08bce865152a47f64ba2b87fd9fd576c7671ebeb79634df2893"
Oct 30 13:45:22.482555 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.482402    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"
Oct 30 13:45:22.486072 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.486055    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"
Oct 30 13:45:22.486216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.486130    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg" event={"ID":"1e0fadc2-ca92-476c-8c21-81f946cfbe21","Type":"ContainerDied","Data":"2789ec66cd609000331da02057ce116e580bd79d9c348aed97a846268e1d4bb3"}
Oct 30 13:45:22.497802 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.497778    4429 scope.go:122] "RemoveContainer" containerID="42d2b9d2933d2441bddd6826780f223ae6c891a56fa4415dcfba21e220d9499e"
Oct 30 13:45:22.554698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.554667    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"]
Oct 30 13:45:22.567808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.567780    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-l6dgf"]
Oct 30 13:45:22.599979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.599950    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"]
Oct 30 13:45:22.603902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:22.603879    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4db150-29889465-jmqmg"]
Oct 30 13:45:24.077187 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:24.077150    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" path="/var/lib/kubelet/pods/1e0fadc2-ca92-476c-8c21-81f946cfbe21/volumes"
Oct 30 13:45:24.077970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:24.077945    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" path="/var/lib/kubelet/pods/a8bad93c-a0e2-4807-821b-08f59773174f/volumes"
Oct 30 13:45:25.163833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:25.163800    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"]
Oct 30 13:45:25.164332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:25.164030    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy" containerID="cri-o://fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77" gracePeriod=30
Oct 30 13:45:30.469317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.469263    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:45:30.560844 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.560808    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-certs\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.560881    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-socket\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.560918    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920d1596-271c-4f07-bd46-061f9adc053d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920d1596-271c-4f07-bd46-061f9adc053d-istiod-ca-cert\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.560998    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-istio-token\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-istio-token\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.561034    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-envoy\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.561063    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/920d1596-271c-4f07-bd46-061f9adc053d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920d1596-271c-4f07-bd46-061f9adc053d-istio-podinfo\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.561094    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-data\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.561219    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-credential-socket\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.561323    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-kube-api-access-f8nd8\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-kube-api-access-f8nd8\") pod \"920d1596-271c-4f07-bd46-061f9adc053d\" (UID: \"920d1596-271c-4f07-bd46-061f9adc053d\") "
Oct 30 13:45:30.561993 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.561942    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-socket" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:30.561993 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.561963    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-certs" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:30.562186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.562105    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.562186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.562129    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.562523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.562369    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-data" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:30.562655 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.562567    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-credential-socket" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:30.564688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.564661    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920d1596-271c-4f07-bd46-061f9adc053d-istiod-ca-cert" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:30.567178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.567126    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/920d1596-271c-4f07-bd46-061f9adc053d-istio-podinfo" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:45:30.567178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.567139    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-envoy" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:45:30.567354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.567217    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-kube-api-access-f8nd8" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "kube-api-access-f8nd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:30.567354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.567250    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-istio-token" pod "920d1596-271c-4f07-bd46-061f9adc053d" (UID: "920d1596-271c-4f07-bd46-061f9adc053d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:30.568961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.568934    4429 generic.go:356] "Generic (PLEG): container finished" podID="920d1596-271c-4f07-bd46-061f9adc053d" containerID="fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77" exitCode=0
Oct 30 13:45:30.569104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.569084    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"
Oct 30 13:45:30.569223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.569080    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" event={"ID":"920d1596-271c-4f07-bd46-061f9adc053d","Type":"ContainerDied","Data":"fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77"}
Oct 30 13:45:30.569391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.569356    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59" event={"ID":"920d1596-271c-4f07-bd46-061f9adc053d","Type":"ContainerDied","Data":"9ac26280e13c69a1d102c9739939a3d76ba1722325381630b79466abff4c93b9"}
Oct 30 13:45:30.569478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.569434    4429 scope.go:122] "RemoveContainer" containerID="fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77"
Oct 30 13:45:30.613236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.613213    4429 scope.go:122] "RemoveContainer" containerID="fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77"
Oct 30 13:45:30.613570 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:30.613549    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77\": container with ID starting with fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77 not found: ID does not exist" containerID="fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77"
Oct 30 13:45:30.613640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.613581    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77"} err="failed to get container status \"fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77\": rpc error: code = NotFound desc = could not find container \"fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77\": container with ID starting with fe50e4b42f452642075e4c99a57f55c102292ebfaadee2dc6d9bb712a4d4cf77 not found: ID does not exist"
Oct 30 13:45:30.652738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.652713    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"]
Oct 30 13:45:30.657700 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.657675    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-22efcf-openshift-default-778dff5c67-zsb59"]
Oct 30 13:45:30.663085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.663061    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.663085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.663085    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8nd8\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-kube-api-access-f8nd8\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.663223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.663097    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/920d1596-271c-4f07-bd46-061f9adc053d-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.663223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.663105    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/920d1596-271c-4f07-bd46-061f9adc053d-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.663223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.663113    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.663223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.663121    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/920d1596-271c-4f07-bd46-061f9adc053d-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.663223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.663128    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/920d1596-271c-4f07-bd46-061f9adc053d-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:30.934960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.934928    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"]
Oct 30 13:45:30.966178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:30.966150    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"]
Oct 30 13:45:32.078644 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:32.078611    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920d1596-271c-4f07-bd46-061f9adc053d" path="/var/lib/kubelet/pods/920d1596-271c-4f07-bd46-061f9adc053d/volumes"
Oct 30 13:45:35.305442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:35.305408    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"]
Oct 30 13:45:35.340600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:35.340570    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"]
Oct 30 13:45:35.936016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:35.935961    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf" podUID="8cf78b02-2d46-480c-9b35-0511616757cf" containerName="payload-processing" containerID="cri-o://6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba" gracePeriod=25
Oct 30 13:45:35.987326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:35.985505    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq" podUID="719a6fc2-d535-4266-b8c7-52c20254c55f" containerName="payload-pre-processing" containerID="cri-o://53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734" gracePeriod=25
Oct 30 13:45:36.166661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.165749    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"]
Oct 30 13:45:36.304137 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.304108    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:45:36.409194 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.409152    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf78b02-2d46-480c-9b35-0511616757cf-kube-api-access-sm9jf\" (UniqueName: \"kubernetes.io/projected/8cf78b02-2d46-480c-9b35-0511616757cf-kube-api-access-sm9jf\") pod \"8cf78b02-2d46-480c-9b35-0511616757cf\" (UID: \"8cf78b02-2d46-480c-9b35-0511616757cf\") "
Oct 30 13:45:36.409737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.409584    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf78b02-2d46-480c-9b35-0511616757cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf78b02-2d46-480c-9b35-0511616757cf-plugins-config-volume\") pod \"8cf78b02-2d46-480c-9b35-0511616757cf\" (UID: \"8cf78b02-2d46-480c-9b35-0511616757cf\") "
Oct 30 13:45:36.409992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.409969    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf78b02-2d46-480c-9b35-0511616757cf-plugins-config-volume" pod "8cf78b02-2d46-480c-9b35-0511616757cf" (UID: "8cf78b02-2d46-480c-9b35-0511616757cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:36.410277 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.410259    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cf78b02-2d46-480c-9b35-0511616757cf-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:36.411662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.411637    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf78b02-2d46-480c-9b35-0511616757cf-kube-api-access-sm9jf" pod "8cf78b02-2d46-480c-9b35-0511616757cf" (UID: "8cf78b02-2d46-480c-9b35-0511616757cf"). InnerVolumeSpecName "kube-api-access-sm9jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:36.511346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.511252    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm9jf\" (UniqueName: \"kubernetes.io/projected/8cf78b02-2d46-480c-9b35-0511616757cf-kube-api-access-sm9jf\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:36.672428 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.672396    4429 generic.go:356] "Generic (PLEG): container finished" podID="8cf78b02-2d46-480c-9b35-0511616757cf" containerID="6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba" exitCode=0
Oct 30 13:45:36.672593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.672450    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"
Oct 30 13:45:36.672593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.672483    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf" event={"ID":"8cf78b02-2d46-480c-9b35-0511616757cf","Type":"ContainerDied","Data":"6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba"}
Oct 30 13:45:36.672593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.672538    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf" event={"ID":"8cf78b02-2d46-480c-9b35-0511616757cf","Type":"ContainerDied","Data":"e2fee697005f089168a11fd339ca4f5195e216c720dff0cd59a0cfa9adeddd98"}
Oct 30 13:45:36.672593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.672563    4429 scope.go:122] "RemoveContainer" containerID="6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba"
Oct 30 13:45:36.685061 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.685038    4429 scope.go:122] "RemoveContainer" containerID="6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba"
Oct 30 13:45:36.685418 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:36.685395    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba\": container with ID starting with 6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba not found: ID does not exist" containerID="6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba"
Oct 30 13:45:36.685497 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.685426    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba"} err="failed to get container status \"6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba\": rpc error: code = NotFound desc = could not find container \"6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba\": container with ID starting with 6fcb70594d075a7d404b72190b542cffbff30a5e685668add04ee187f506c5ba not found: ID does not exist"
Oct 30 13:45:36.731638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.731607    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"]
Oct 30 13:45:36.735779 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.735754    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-025c7f-f8b9959c4-s4rtf"]
Oct 30 13:45:36.777751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:36.777632    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq" podUID="719a6fc2-d535-4266-b8c7-52c20254c55f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:38.084164 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:38.084128    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf78b02-2d46-480c-9b35-0511616757cf" path="/var/lib/kubelet/pods/8cf78b02-2d46-480c-9b35-0511616757cf/volumes"
Oct 30 13:45:40.305713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:40.305665    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g" podUID="2c64c968-129c-4f4e-806e-a01a1a59e267" containerName="payload-processing" containerID="cri-o://8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615" gracePeriod=25
Oct 30 13:45:40.341457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:40.341411    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz" podUID="0af1c0aa-06d3-401e-94ce-021effe0ba5f" containerName="payload-pre-processing" containerID="cri-o://08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf" gracePeriod=25
Oct 30 13:45:40.373073 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:40.373038    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"]
Oct 30 13:45:40.373372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:40.373345    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9" podUID="35af2313-f69b-4be2-ba2e-eb9987eeae3b" containerName="maas-api" containerID="cri-o://0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2" gracePeriod=30
Oct 30 13:45:41.166922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.166854    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d" podUID="efa69d05-7642-40a4-8a81-ae56b7952142" containerName="payload-pre-processing" containerID="cri-o://445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba" gracePeriod=25
Oct 30 13:45:41.676225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.676194    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:45:41.720578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.720501    4429 generic.go:356] "Generic (PLEG): container finished" podID="35af2313-f69b-4be2-ba2e-eb9987eeae3b" containerID="0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2" exitCode=0
Oct 30 13:45:41.720578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.720556    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"
Oct 30 13:45:41.720578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.720571    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9" event={"ID":"35af2313-f69b-4be2-ba2e-eb9987eeae3b","Type":"ContainerDied","Data":"0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2"}
Oct 30 13:45:41.720811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.720635    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9" event={"ID":"35af2313-f69b-4be2-ba2e-eb9987eeae3b","Type":"ContainerDied","Data":"1245057192ee4b243450f7744ca747e023463ef80a85bcc14b0a6ab991aa9c9f"}
Oct 30 13:45:41.720811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.720668    4429 scope.go:122] "RemoveContainer" containerID="0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2"
Oct 30 13:45:41.740211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.735571    4429 scope.go:122] "RemoveContainer" containerID="0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2"
Oct 30 13:45:41.740211 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:41.735963    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2\": container with ID starting with 0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2 not found: ID does not exist" containerID="0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2"
Oct 30 13:45:41.740211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.735996    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2"} err="failed to get container status \"0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2\": rpc error: code = NotFound desc = could not find container \"0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2\": container with ID starting with 0fb6c37d1210207f99b9f674b0764746a8458c2c1b24229816b7eafcf03051d2 not found: ID does not exist"
Oct 30 13:45:41.794272 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.794239    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35af2313-f69b-4be2-ba2e-eb9987eeae3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af2313-f69b-4be2-ba2e-eb9987eeae3b-openshift-service-ca\") pod \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") "
Oct 30 13:45:41.794460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.794382    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-metrics-certs\") pod \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") "
Oct 30 13:45:41.794557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.794536    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-maas-api-tls\") pod \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") "
Oct 30 13:45:41.794645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.794629    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35af2313-f69b-4be2-ba2e-eb9987eeae3b-kube-api-access-ct5jm\" (UniqueName: \"kubernetes.io/projected/35af2313-f69b-4be2-ba2e-eb9987eeae3b-kube-api-access-ct5jm\") pod \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\" (UID: \"35af2313-f69b-4be2-ba2e-eb9987eeae3b\") "
Oct 30 13:45:41.794704 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.794683    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35af2313-f69b-4be2-ba2e-eb9987eeae3b-openshift-service-ca" pod "35af2313-f69b-4be2-ba2e-eb9987eeae3b" (UID: "35af2313-f69b-4be2-ba2e-eb9987eeae3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:41.795277 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.795227    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af2313-f69b-4be2-ba2e-eb9987eeae3b-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:41.796719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.796690    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-metrics-certs" pod "35af2313-f69b-4be2-ba2e-eb9987eeae3b" (UID: "35af2313-f69b-4be2-ba2e-eb9987eeae3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:41.796847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.796805    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-maas-api-tls" pod "35af2313-f69b-4be2-ba2e-eb9987eeae3b" (UID: "35af2313-f69b-4be2-ba2e-eb9987eeae3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:41.796847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.796830    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35af2313-f69b-4be2-ba2e-eb9987eeae3b-kube-api-access-ct5jm" pod "35af2313-f69b-4be2-ba2e-eb9987eeae3b" (UID: "35af2313-f69b-4be2-ba2e-eb9987eeae3b"). InnerVolumeSpecName "kube-api-access-ct5jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:41.896760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.896728    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:41.896760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.896758    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct5jm\" (UniqueName: \"kubernetes.io/projected/35af2313-f69b-4be2-ba2e-eb9987eeae3b-kube-api-access-ct5jm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:41.896760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:41.896768    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35af2313-f69b-4be2-ba2e-eb9987eeae3b-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:42.082821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:42.082793    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"]
Oct 30 13:45:42.089032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:42.089001    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-025c7f-5bf85b8cf4-r7bb9"]
Oct 30 13:45:42.374779 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:42.374691    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:45:43.856364 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:43.856325    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g" podUID="2c64c968-129c-4f4e-806e-a01a1a59e267" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:44.077936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:44.077904    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35af2313-f69b-4be2-ba2e-eb9987eeae3b" path="/var/lib/kubelet/pods/35af2313-f69b-4be2-ba2e-eb9987eeae3b/volumes"
Oct 30 13:45:44.864128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:44.864081    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz" podUID="0af1c0aa-06d3-401e-94ce-021effe0ba5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:46.123587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:46.123549    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d" podUID="efa69d05-7642-40a4-8a81-ae56b7952142" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:46.776384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:46.776347    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq" podUID="719a6fc2-d535-4266-b8c7-52c20254c55f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:53.856676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:53.856634    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g" podUID="2c64c968-129c-4f4e-806e-a01a1a59e267" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:54.863956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:54.863909    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz" podUID="0af1c0aa-06d3-401e-94ce-021effe0ba5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:56.132318 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.124391    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d" podUID="efa69d05-7642-40a4-8a81-ae56b7952142" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:45:56.193222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.193197    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:45:56.248748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.248678    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719a6fc2-d535-4266-b8c7-52c20254c55f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719a6fc2-d535-4266-b8c7-52c20254c55f-plugins-config-volume\") pod \"719a6fc2-d535-4266-b8c7-52c20254c55f\" (UID: \"719a6fc2-d535-4266-b8c7-52c20254c55f\") "
Oct 30 13:45:56.248748 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.248736    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719a6fc2-d535-4266-b8c7-52c20254c55f-kube-api-access-p76vv\" (UniqueName: \"kubernetes.io/projected/719a6fc2-d535-4266-b8c7-52c20254c55f-kube-api-access-p76vv\") pod \"719a6fc2-d535-4266-b8c7-52c20254c55f\" (UID: \"719a6fc2-d535-4266-b8c7-52c20254c55f\") "
Oct 30 13:45:56.249070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.249043    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719a6fc2-d535-4266-b8c7-52c20254c55f-plugins-config-volume" pod "719a6fc2-d535-4266-b8c7-52c20254c55f" (UID: "719a6fc2-d535-4266-b8c7-52c20254c55f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:56.249185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.249170    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719a6fc2-d535-4266-b8c7-52c20254c55f-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:56.251015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.250989    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719a6fc2-d535-4266-b8c7-52c20254c55f-kube-api-access-p76vv" pod "719a6fc2-d535-4266-b8c7-52c20254c55f" (UID: "719a6fc2-d535-4266-b8c7-52c20254c55f"). InnerVolumeSpecName "kube-api-access-p76vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:56.350576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.350532    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76vv\" (UniqueName: \"kubernetes.io/projected/719a6fc2-d535-4266-b8c7-52c20254c55f-kube-api-access-p76vv\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:45:56.877593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.877555    4429 generic.go:356] "Generic (PLEG): container finished" podID="719a6fc2-d535-4266-b8c7-52c20254c55f" containerID="53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734" exitCode=0
Oct 30 13:45:56.877801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.877626    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"
Oct 30 13:45:56.877801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.877653    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq" event={"ID":"719a6fc2-d535-4266-b8c7-52c20254c55f","Type":"ContainerDied","Data":"53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734"}
Oct 30 13:45:56.877801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.877723    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq" event={"ID":"719a6fc2-d535-4266-b8c7-52c20254c55f","Type":"ContainerDied","Data":"d21b49b34e0a73175850985bcf6df37e3c70b84621d423d2304d40610a628e7e"}
Oct 30 13:45:56.877801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.877753    4429 scope.go:122] "RemoveContainer" containerID="53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734"
Oct 30 13:45:56.889392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.889279    4429 scope.go:122] "RemoveContainer" containerID="53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734"
Oct 30 13:45:56.889680 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:45:56.889662    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734\": container with ID starting with 53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734 not found: ID does not exist" containerID="53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734"
Oct 30 13:45:56.889731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.889687    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734"} err="failed to get container status \"53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734\": rpc error: code = NotFound desc = could not find container \"53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734\": container with ID starting with 53d0fcf0d28431397c24ba1851f00ab385b406aa718892f14e55b397e8bfc734 not found: ID does not exist"
Oct 30 13:45:56.939231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.939202    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"]
Oct 30 13:45:56.952983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:56.952958    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-025c7f-59d495bd6c-75xkq"]
Oct 30 13:45:57.448586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:57.448550    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"]
Oct 30 13:45:58.081995 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:45:58.081951    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719a6fc2-d535-4266-b8c7-52c20254c55f" path="/var/lib/kubelet/pods/719a6fc2-d535-4266-b8c7-52c20254c55f/volumes"
Oct 30 13:46:00.620669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.620637    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"]
Oct 30 13:46:00.649064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.649028    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:46:00.658923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.658895    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"]
Oct 30 13:46:00.687088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.687059    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:46:00.729234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.729199    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c64c968-129c-4f4e-806e-a01a1a59e267-kube-api-access-vhzx4\" (UniqueName: \"kubernetes.io/projected/2c64c968-129c-4f4e-806e-a01a1a59e267-kube-api-access-vhzx4\") pod \"2c64c968-129c-4f4e-806e-a01a1a59e267\" (UID: \"2c64c968-129c-4f4e-806e-a01a1a59e267\") "
Oct 30 13:46:00.729461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.729261    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c64c968-129c-4f4e-806e-a01a1a59e267-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c64c968-129c-4f4e-806e-a01a1a59e267-plugins-config-volume\") pod \"2c64c968-129c-4f4e-806e-a01a1a59e267\" (UID: \"2c64c968-129c-4f4e-806e-a01a1a59e267\") "
Oct 30 13:46:00.730511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.730486    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c64c968-129c-4f4e-806e-a01a1a59e267-plugins-config-volume" pod "2c64c968-129c-4f4e-806e-a01a1a59e267" (UID: "2c64c968-129c-4f4e-806e-a01a1a59e267"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:00.736644 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.736610    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c64c968-129c-4f4e-806e-a01a1a59e267-kube-api-access-vhzx4" pod "2c64c968-129c-4f4e-806e-a01a1a59e267" (UID: "2c64c968-129c-4f4e-806e-a01a1a59e267"). InnerVolumeSpecName "kube-api-access-vhzx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:00.831594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.831505    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af1c0aa-06d3-401e-94ce-021effe0ba5f-kube-api-access-cd68q\" (UniqueName: \"kubernetes.io/projected/0af1c0aa-06d3-401e-94ce-021effe0ba5f-kube-api-access-cd68q\") pod \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\" (UID: \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\") "
Oct 30 13:46:00.831779 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.831653    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af1c0aa-06d3-401e-94ce-021effe0ba5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af1c0aa-06d3-401e-94ce-021effe0ba5f-plugins-config-volume\") pod \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\" (UID: \"0af1c0aa-06d3-401e-94ce-021effe0ba5f\") "
Oct 30 13:46:00.832043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.832020    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af1c0aa-06d3-401e-94ce-021effe0ba5f-plugins-config-volume" pod "0af1c0aa-06d3-401e-94ce-021effe0ba5f" (UID: "0af1c0aa-06d3-401e-94ce-021effe0ba5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:00.832180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.832165    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhzx4\" (UniqueName: \"kubernetes.io/projected/2c64c968-129c-4f4e-806e-a01a1a59e267-kube-api-access-vhzx4\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:00.832253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.832182    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c64c968-129c-4f4e-806e-a01a1a59e267-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:00.832253 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.832192    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af1c0aa-06d3-401e-94ce-021effe0ba5f-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:00.833940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.833914    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af1c0aa-06d3-401e-94ce-021effe0ba5f-kube-api-access-cd68q" pod "0af1c0aa-06d3-401e-94ce-021effe0ba5f" (UID: "0af1c0aa-06d3-401e-94ce-021effe0ba5f"). InnerVolumeSpecName "kube-api-access-cd68q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:00.912107 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.912068    4429 generic.go:356] "Generic (PLEG): container finished" podID="2c64c968-129c-4f4e-806e-a01a1a59e267" containerID="8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615" exitCode=0
Oct 30 13:46:00.912371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.912121    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"
Oct 30 13:46:00.912371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.912154    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g" event={"ID":"2c64c968-129c-4f4e-806e-a01a1a59e267","Type":"ContainerDied","Data":"8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615"}
Oct 30 13:46:00.912371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.912220    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g" event={"ID":"2c64c968-129c-4f4e-806e-a01a1a59e267","Type":"ContainerDied","Data":"838b3884fd3a4e3abe0df348f612a9799ed0e6fa7afc6cfe7c63600ca4c17b39"}
Oct 30 13:46:00.912371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.912252    4429 scope.go:122] "RemoveContainer" containerID="8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615"
Oct 30 13:46:00.913921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.913898    4429 generic.go:356] "Generic (PLEG): container finished" podID="0af1c0aa-06d3-401e-94ce-021effe0ba5f" containerID="08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf" exitCode=0
Oct 30 13:46:00.914033 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.913947    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"
Oct 30 13:46:00.914033 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.913987    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz" event={"ID":"0af1c0aa-06d3-401e-94ce-021effe0ba5f","Type":"ContainerDied","Data":"08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf"}
Oct 30 13:46:00.914143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.914047    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz" event={"ID":"0af1c0aa-06d3-401e-94ce-021effe0ba5f","Type":"ContainerDied","Data":"79ec57936361f55d1bf17d16f4fa6d21ca646259b8d110f1ce89213fe541c3ae"}
Oct 30 13:46:00.926023 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.926001    4429 scope.go:122] "RemoveContainer" containerID="8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615"
Oct 30 13:46:00.926465 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:00.926428    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615\": container with ID starting with 8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615 not found: ID does not exist" containerID="8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615"
Oct 30 13:46:00.926561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.926466    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615"} err="failed to get container status \"8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615\": rpc error: code = NotFound desc = could not find container \"8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615\": container with ID starting with 8e179a320a991860f9e03c71bd6d5f626c0bfb36c1347d4626226d2887c7a615 not found: ID does not exist"
Oct 30 13:46:00.926561 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.926483    4429 scope.go:122] "RemoveContainer" containerID="08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf"
Oct 30 13:46:00.934735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.934709    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd68q\" (UniqueName: \"kubernetes.io/projected/0af1c0aa-06d3-401e-94ce-021effe0ba5f-kube-api-access-cd68q\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:00.937692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.937672    4429 scope.go:122] "RemoveContainer" containerID="08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf"
Oct 30 13:46:00.937960 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:00.937936    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf\": container with ID starting with 08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf not found: ID does not exist" containerID="08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf"
Oct 30 13:46:00.938004 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.937968    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf"} err="failed to get container status \"08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf\": rpc error: code = NotFound desc = could not find container \"08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf\": container with ID starting with 08191fca8f601d7c6b96f37580436d969c98b018a1a06be3c1ca6108dbea91bf not found: ID does not exist"
Oct 30 13:46:00.974378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.974346    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"]
Oct 30 13:46:00.977569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:00.977537    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09ad1e-5649d5847-qxxzz"]
Oct 30 13:46:01.018797 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.018765    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"]
Oct 30 13:46:01.025469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.025442    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09ad1e-b6fbd8696-jpz2g"]
Oct 30 13:46:01.283263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.283223    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"]
Oct 30 13:46:01.283543 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.283512    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy" containerID="cri-o://7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523" gracePeriod=30
Oct 30 13:46:01.579054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.579029    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:46:01.675026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.674984    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa69d05-7642-40a4-8a81-ae56b7952142-kube-api-access-kjq45\" (UniqueName: \"kubernetes.io/projected/efa69d05-7642-40a4-8a81-ae56b7952142-kube-api-access-kjq45\") pod \"efa69d05-7642-40a4-8a81-ae56b7952142\" (UID: \"efa69d05-7642-40a4-8a81-ae56b7952142\") "
Oct 30 13:46:01.675617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.675366    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa69d05-7642-40a4-8a81-ae56b7952142-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa69d05-7642-40a4-8a81-ae56b7952142-plugins-config-volume\") pod \"efa69d05-7642-40a4-8a81-ae56b7952142\" (UID: \"efa69d05-7642-40a4-8a81-ae56b7952142\") "
Oct 30 13:46:01.676420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.676388    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa69d05-7642-40a4-8a81-ae56b7952142-plugins-config-volume" pod "efa69d05-7642-40a4-8a81-ae56b7952142" (UID: "efa69d05-7642-40a4-8a81-ae56b7952142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:01.682717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.682686    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa69d05-7642-40a4-8a81-ae56b7952142-kube-api-access-kjq45" pod "efa69d05-7642-40a4-8a81-ae56b7952142" (UID: "efa69d05-7642-40a4-8a81-ae56b7952142"). InnerVolumeSpecName "kube-api-access-kjq45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:01.778186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.778152    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa69d05-7642-40a4-8a81-ae56b7952142-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:01.778186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.778187    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjq45\" (UniqueName: \"kubernetes.io/projected/efa69d05-7642-40a4-8a81-ae56b7952142-kube-api-access-kjq45\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:01.923724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.923637    4429 generic.go:356] "Generic (PLEG): container finished" podID="efa69d05-7642-40a4-8a81-ae56b7952142" containerID="445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba" exitCode=0
Oct 30 13:46:01.923724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.923695    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"
Oct 30 13:46:01.923998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.923735    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d" event={"ID":"efa69d05-7642-40a4-8a81-ae56b7952142","Type":"ContainerDied","Data":"445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba"}
Oct 30 13:46:01.923998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.923799    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d" event={"ID":"efa69d05-7642-40a4-8a81-ae56b7952142","Type":"ContainerDied","Data":"df6ac0e0b18752e850bdfde030b11b191433a8bebd19e0894d0bb563cc6ed8d1"}
Oct 30 13:46:01.923998 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.923834    4429 scope.go:122] "RemoveContainer" containerID="445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba"
Oct 30 13:46:01.948807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.948781    4429 scope.go:122] "RemoveContainer" containerID="445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba"
Oct 30 13:46:01.949221 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:01.949196    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba\": container with ID starting with 445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba not found: ID does not exist" containerID="445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba"
Oct 30 13:46:01.949362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:01.949231    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba"} err="failed to get container status \"445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba\": rpc error: code = NotFound desc = could not find container \"445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba\": container with ID starting with 445e70ea3c14e60447c3bbbbd2353727133efd71b11623562393df118c6180ba not found: ID does not exist"
Oct 30 13:46:02.001393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:02.001361    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"]
Oct 30 13:46:02.004074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:02.004046    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b7407-c548f9455-dth8d"]
Oct 30 13:46:02.078070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:02.078039    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af1c0aa-06d3-401e-94ce-021effe0ba5f" path="/var/lib/kubelet/pods/0af1c0aa-06d3-401e-94ce-021effe0ba5f/volumes"
Oct 30 13:46:02.078506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:02.078491    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c64c968-129c-4f4e-806e-a01a1a59e267" path="/var/lib/kubelet/pods/2c64c968-129c-4f4e-806e-a01a1a59e267/volumes"
Oct 30 13:46:02.078877 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:02.078864    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa69d05-7642-40a4-8a81-ae56b7952142" path="/var/lib/kubelet/pods/efa69d05-7642-40a4-8a81-ae56b7952142/volumes"
Oct 30 13:46:02.449437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:02.449389    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r" podUID="1e68bd2b-8c5d-462a-a91d-c6ba92965718" containerName="payload-processing" containerID="cri-o://900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860" gracePeriod=25
Oct 30 13:46:05.621872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.621825    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks" podUID="4e4761cc-99df-4415-9b05-153a79dfc5fc" containerName="payload-processing" containerID="cri-o://7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c" gracePeriod=25
Oct 30 13:46:05.659466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.659422    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp" podUID="f3b6e91c-2f71-4503-b75c-4dba639ae8d7" containerName="payload-pre-processing" containerID="cri-o://87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345" gracePeriod=25
Oct 30 13:46:05.912544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.912505    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:46:05.952096 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.952061    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4761cc-99df-4415-9b05-153a79dfc5fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4761cc-99df-4415-9b05-153a79dfc5fc-plugins-config-volume\") pod \"4e4761cc-99df-4415-9b05-153a79dfc5fc\" (UID: \"4e4761cc-99df-4415-9b05-153a79dfc5fc\") "
Oct 30 13:46:05.952515 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.952494    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4761cc-99df-4415-9b05-153a79dfc5fc-plugins-config-volume" pod "4e4761cc-99df-4415-9b05-153a79dfc5fc" (UID: "4e4761cc-99df-4415-9b05-153a79dfc5fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:05.952586 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.952524    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4761cc-99df-4415-9b05-153a79dfc5fc-kube-api-access-qwvmq\" (UniqueName: \"kubernetes.io/projected/4e4761cc-99df-4415-9b05-153a79dfc5fc-kube-api-access-qwvmq\") pod \"4e4761cc-99df-4415-9b05-153a79dfc5fc\" (UID: \"4e4761cc-99df-4415-9b05-153a79dfc5fc\") "
Oct 30 13:46:05.952957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.952941    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4761cc-99df-4415-9b05-153a79dfc5fc-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:05.954885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.954861    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4761cc-99df-4415-9b05-153a79dfc5fc-kube-api-access-qwvmq" pod "4e4761cc-99df-4415-9b05-153a79dfc5fc" (UID: "4e4761cc-99df-4415-9b05-153a79dfc5fc"). InnerVolumeSpecName "kube-api-access-qwvmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:05.977038 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.976998    4429 generic.go:356] "Generic (PLEG): container finished" podID="4e4761cc-99df-4415-9b05-153a79dfc5fc" containerID="7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c" exitCode=0
Oct 30 13:46:05.977254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.977058    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"
Oct 30 13:46:05.977254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.977116    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks" event={"ID":"4e4761cc-99df-4415-9b05-153a79dfc5fc","Type":"ContainerDied","Data":"7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c"}
Oct 30 13:46:05.977254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.977175    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks" event={"ID":"4e4761cc-99df-4415-9b05-153a79dfc5fc","Type":"ContainerDied","Data":"76e3264f3315c9ca6b37c04227a1433adf4e10f1962607ea617c40043d87e0e9"}
Oct 30 13:46:05.977254 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.977208    4429 scope.go:122] "RemoveContainer" containerID="7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c"
Oct 30 13:46:05.992778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.992751    4429 scope.go:122] "RemoveContainer" containerID="7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c"
Oct 30 13:46:05.993217 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:05.993185    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c\": container with ID starting with 7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c not found: ID does not exist" containerID="7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c"
Oct 30 13:46:05.993327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:05.993230    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c"} err="failed to get container status \"7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c\": rpc error: code = NotFound desc = could not find container \"7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c\": container with ID starting with 7e8d34506936f4c80bd9ff38c0d762fca3d13e1c5fd1b3c15656587f4802703c not found: ID does not exist"
Oct 30 13:46:06.036849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.036815    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"]
Oct 30 13:46:06.043943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.043913    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f97234-69b66665f6-78mks"]
Oct 30 13:46:06.054308 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.054255    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwvmq\" (UniqueName: \"kubernetes.io/projected/4e4761cc-99df-4415-9b05-153a79dfc5fc-kube-api-access-qwvmq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.077091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.077049    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4761cc-99df-4415-9b05-153a79dfc5fc" path="/var/lib/kubelet/pods/4e4761cc-99df-4415-9b05-153a79dfc5fc/volumes"
Oct 30 13:46:06.550506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.550433    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:46:06.662905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.662867    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-envoy\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.663437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.662945    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-kube-api-access-d87bx\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-kube-api-access-d87bx\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.663437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.662993    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-certs\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.663437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.663124    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-podinfo\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.663437 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.663216    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-socket\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.663696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.663429    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-certs" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.663696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.663475    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-token\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-token\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.663696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.663559    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-credential-socket\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.663696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.663608    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-data\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.663696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.663645    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istiod-ca-cert\") pod \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\" (UID: \"ce57651f-cec8-4d1b-b46d-04b081bd0ac8\") "
Oct 30 13:46:06.665621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.664357    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.665621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.664819    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istiod-ca-cert" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:06.665621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.665157    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-credential-socket" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.665621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.665460    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-data" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.665621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.665583    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-socket" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.666562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.666514    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-podinfo" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:06.668511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.668479    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-token" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:06.668758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.668733    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-envoy" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:06.669904 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.669880    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-kube-api-access-d87bx" pod "ce57651f-cec8-4d1b-b46d-04b081bd0ac8" (UID: "ce57651f-cec8-4d1b-b46d-04b081bd0ac8"). InnerVolumeSpecName "kube-api-access-d87bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:06.765323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.765277    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.765323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.765320    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d87bx\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-kube-api-access-d87bx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.765323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.765332    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.765577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.765342    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.765577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.765352    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.765577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.765362    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.765577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.765370    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.765577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.765378    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce57651f-cec8-4d1b-b46d-04b081bd0ac8-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.988456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.988420    4429 generic.go:356] "Generic (PLEG): container finished" podID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerID="7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523" exitCode=0
Oct 30 13:46:06.988669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.988470    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"
Oct 30 13:46:06.988669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.988502    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" event={"ID":"ce57651f-cec8-4d1b-b46d-04b081bd0ac8","Type":"ContainerDied","Data":"7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523"}
Oct 30 13:46:06.988669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.988565    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h" event={"ID":"ce57651f-cec8-4d1b-b46d-04b081bd0ac8","Type":"ContainerDied","Data":"32d22b4e8332711b06c3a76ceb63215c9dea4f55b265e04d3a6bf641d05e5e96"}
Oct 30 13:46:06.988669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:06.988594    4429 scope.go:122] "RemoveContainer" containerID="7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523"
Oct 30 13:46:07.008008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:07.007980    4429 scope.go:122] "RemoveContainer" containerID="7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523"
Oct 30 13:46:07.008395 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:07.008370    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523\": container with ID starting with 7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523 not found: ID does not exist" containerID="7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523"
Oct 30 13:46:07.008471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:07.008409    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523"} err="failed to get container status \"7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523\": rpc error: code = NotFound desc = could not find container \"7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523\": container with ID starting with 7765ca70e587e19354322a7dfecf6bc9f046ff4e6ae05c7c940bd47e50597523 not found: ID does not exist"
Oct 30 13:46:07.055176 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:07.055133    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"]
Oct 30 13:46:07.068528 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:07.068487    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-09ad1e-openshift-default-5845b78b67-jg55h"]
Oct 30 13:46:08.104310 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:08.084407    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" path="/var/lib/kubelet/pods/ce57651f-cec8-4d1b-b46d-04b081bd0ac8/volumes"
Oct 30 13:46:10.876366 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:10.876328    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp" podUID="f3b6e91c-2f71-4503-b75c-4dba639ae8d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:46:11.819839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:11.819800    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r" podUID="1e68bd2b-8c5d-462a-a91d-c6ba92965718" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:46:12.357132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:12.357093    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"]
Oct 30 13:46:12.375312 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:12.375264    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:46:12.392660 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:12.392631    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"]
Oct 30 13:46:17.357832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.357782    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9" podUID="b8dcaf96-99c3-42c7-8e03-4f45e4433287" containerName="payload-processing" containerID="cri-o://74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b" gracePeriod=25
Oct 30 13:46:17.393261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.393222    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8" podUID="dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" containerName="payload-pre-processing" containerID="cri-o://a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c" gracePeriod=25
Oct 30 13:46:17.692711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.691780    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:46:17.769937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.769903    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8dcaf96-99c3-42c7-8e03-4f45e4433287-kube-api-access-tsshv\" (UniqueName: \"kubernetes.io/projected/b8dcaf96-99c3-42c7-8e03-4f45e4433287-kube-api-access-tsshv\") pod \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\" (UID: \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\") "
Oct 30 13:46:17.772251 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.772219    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dcaf96-99c3-42c7-8e03-4f45e4433287-kube-api-access-tsshv" pod "b8dcaf96-99c3-42c7-8e03-4f45e4433287" (UID: "b8dcaf96-99c3-42c7-8e03-4f45e4433287"). InnerVolumeSpecName "kube-api-access-tsshv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:17.870800 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.870759    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8dcaf96-99c3-42c7-8e03-4f45e4433287-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8dcaf96-99c3-42c7-8e03-4f45e4433287-plugins-config-volume\") pod \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\" (UID: \"b8dcaf96-99c3-42c7-8e03-4f45e4433287\") "
Oct 30 13:46:17.871139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.871112    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dcaf96-99c3-42c7-8e03-4f45e4433287-plugins-config-volume" pod "b8dcaf96-99c3-42c7-8e03-4f45e4433287" (UID: "b8dcaf96-99c3-42c7-8e03-4f45e4433287"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:17.871495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.871479    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8dcaf96-99c3-42c7-8e03-4f45e4433287-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:17.871546 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:17.871498    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsshv\" (UniqueName: \"kubernetes.io/projected/b8dcaf96-99c3-42c7-8e03-4f45e4433287-kube-api-access-tsshv\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:18.116369 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.116325    4429 generic.go:356] "Generic (PLEG): container finished" podID="b8dcaf96-99c3-42c7-8e03-4f45e4433287" containerID="74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b" exitCode=0
Oct 30 13:46:18.116551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.116384    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"
Oct 30 13:46:18.116551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.116404    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9" event={"ID":"b8dcaf96-99c3-42c7-8e03-4f45e4433287","Type":"ContainerDied","Data":"74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b"}
Oct 30 13:46:18.116551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.116468    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9" event={"ID":"b8dcaf96-99c3-42c7-8e03-4f45e4433287","Type":"ContainerDied","Data":"7639e599a882a49bb3189fcbba45b5ce7e0ed0a299f46f17d7738693532a4a6b"}
Oct 30 13:46:18.116551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.116497    4429 scope.go:122] "RemoveContainer" containerID="74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b"
Oct 30 13:46:18.127600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.127581    4429 scope.go:122] "RemoveContainer" containerID="74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b"
Oct 30 13:46:18.127918 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:18.127893    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b\": container with ID starting with 74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b not found: ID does not exist" containerID="74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b"
Oct 30 13:46:18.127979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.127929    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b"} err="failed to get container status \"74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b\": rpc error: code = NotFound desc = could not find container \"74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b\": container with ID starting with 74f99a8446181a9869288b6941f7a40f9fd04787746a5d80f79413e1c5dd2a3b not found: ID does not exist"
Oct 30 13:46:18.180171 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.180139    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"]
Oct 30 13:46:18.183959 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:18.183933    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8369-746d5899b8-zg9t9"]
Oct 30 13:46:20.077735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:20.077704    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8dcaf96-99c3-42c7-8e03-4f45e4433287" path="/var/lib/kubelet/pods/b8dcaf96-99c3-42c7-8e03-4f45e4433287/volumes"
Oct 30 13:46:20.876378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:20.876332    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp" podUID="f3b6e91c-2f71-4503-b75c-4dba639ae8d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:46:21.819622 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:21.819563    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r" podUID="1e68bd2b-8c5d-462a-a91d-c6ba92965718" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:46:22.744102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:22.744077    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:46:22.847618 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:22.847585    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e68bd2b-8c5d-462a-a91d-c6ba92965718-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68bd2b-8c5d-462a-a91d-c6ba92965718-plugins-config-volume\") pod \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\" (UID: \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\") "
Oct 30 13:46:22.848132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:22.847888    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e68bd2b-8c5d-462a-a91d-c6ba92965718-kube-api-access-jqd45\" (UniqueName: \"kubernetes.io/projected/1e68bd2b-8c5d-462a-a91d-c6ba92965718-kube-api-access-jqd45\") pod \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\" (UID: \"1e68bd2b-8c5d-462a-a91d-c6ba92965718\") "
Oct 30 13:46:22.848132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:22.848030    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e68bd2b-8c5d-462a-a91d-c6ba92965718-plugins-config-volume" pod "1e68bd2b-8c5d-462a-a91d-c6ba92965718" (UID: "1e68bd2b-8c5d-462a-a91d-c6ba92965718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:22.848455 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:22.848430    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e68bd2b-8c5d-462a-a91d-c6ba92965718-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:22.850177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:22.850155    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e68bd2b-8c5d-462a-a91d-c6ba92965718-kube-api-access-jqd45" pod "1e68bd2b-8c5d-462a-a91d-c6ba92965718" (UID: "1e68bd2b-8c5d-462a-a91d-c6ba92965718"). InnerVolumeSpecName "kube-api-access-jqd45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:22.949705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:22.949669    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqd45\" (UniqueName: \"kubernetes.io/projected/1e68bd2b-8c5d-462a-a91d-c6ba92965718-kube-api-access-jqd45\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:23.016926 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.016834    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8" podUID="dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:46:23.159593 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.159562    4429 generic.go:356] "Generic (PLEG): container finished" podID="1e68bd2b-8c5d-462a-a91d-c6ba92965718" containerID="900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860" exitCode=0
Oct 30 13:46:23.159769 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.159614    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"
Oct 30 13:46:23.159769 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.159649    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r" event={"ID":"1e68bd2b-8c5d-462a-a91d-c6ba92965718","Type":"ContainerDied","Data":"900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860"}
Oct 30 13:46:23.159769 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.159711    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r" event={"ID":"1e68bd2b-8c5d-462a-a91d-c6ba92965718","Type":"ContainerDied","Data":"8a970fc79ee6ea94a681297f3f6583ef27f73a0e0125eea0ada5f567fb8cc3e8"}
Oct 30 13:46:23.159769 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.159743    4429 scope.go:122] "RemoveContainer" containerID="900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860"
Oct 30 13:46:23.171637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.171618    4429 scope.go:122] "RemoveContainer" containerID="900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860"
Oct 30 13:46:23.171962 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:23.171944    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860\": container with ID starting with 900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860 not found: ID does not exist" containerID="900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860"
Oct 30 13:46:23.172003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.171971    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860"} err="failed to get container status \"900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860\": rpc error: code = NotFound desc = could not find container \"900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860\": container with ID starting with 900287c34c38552a57d3b2cf8abf2591d7bad5a9a310a6857551b0b856fde860 not found: ID does not exist"
Oct 30 13:46:23.219225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.219189    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"]
Oct 30 13:46:23.223676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:23.223649    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9a93bd-bdbb476fd-cmm5r"]
Oct 30 13:46:24.078821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:24.078784    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e68bd2b-8c5d-462a-a91d-c6ba92965718" path="/var/lib/kubelet/pods/1e68bd2b-8c5d-462a-a91d-c6ba92965718/volumes"
Oct 30 13:46:25.945826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:25.945794    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:46:25.997079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:25.997042    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-plugins-config-volume\") pod \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\" (UID: \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\") "
Oct 30 13:46:25.997366 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:25.997317    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-kube-api-access-x8vs7\" (UniqueName: \"kubernetes.io/projected/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-kube-api-access-x8vs7\") pod \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\" (UID: \"f3b6e91c-2f71-4503-b75c-4dba639ae8d7\") "
Oct 30 13:46:25.999168 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:25.999140    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-plugins-config-volume" pod "f3b6e91c-2f71-4503-b75c-4dba639ae8d7" (UID: "f3b6e91c-2f71-4503-b75c-4dba639ae8d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:26.001325 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.001277    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-kube-api-access-x8vs7" pod "f3b6e91c-2f71-4503-b75c-4dba639ae8d7" (UID: "f3b6e91c-2f71-4503-b75c-4dba639ae8d7"). InnerVolumeSpecName "kube-api-access-x8vs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:26.099825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.099795    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:26.099825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.099820    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8vs7\" (UniqueName: \"kubernetes.io/projected/f3b6e91c-2f71-4503-b75c-4dba639ae8d7-kube-api-access-x8vs7\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:26.186122 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.186086    4429 generic.go:356] "Generic (PLEG): container finished" podID="f3b6e91c-2f71-4503-b75c-4dba639ae8d7" containerID="87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345" exitCode=0
Oct 30 13:46:26.186353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.186140    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"
Oct 30 13:46:26.186353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.186168    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp" event={"ID":"f3b6e91c-2f71-4503-b75c-4dba639ae8d7","Type":"ContainerDied","Data":"87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345"}
Oct 30 13:46:26.186353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.186240    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp" event={"ID":"f3b6e91c-2f71-4503-b75c-4dba639ae8d7","Type":"ContainerDied","Data":"14da98adff52ec7f861095190c06bf7ea163abce1be39e287493e9e89cfa0312"}
Oct 30 13:46:26.186353 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.186269    4429 scope.go:122] "RemoveContainer" containerID="87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345"
Oct 30 13:46:26.197324 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.197265    4429 scope.go:122] "RemoveContainer" containerID="87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345"
Oct 30 13:46:26.197698 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:26.197675    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345\": container with ID starting with 87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345 not found: ID does not exist" containerID="87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345"
Oct 30 13:46:26.197816 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.197707    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345"} err="failed to get container status \"87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345\": rpc error: code = NotFound desc = could not find container \"87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345\": container with ID starting with 87288b9352f12eb521ef183fd8f08ff0ee830add0b754fdddb5676db224cd345 not found: ID does not exist"
Oct 30 13:46:26.247430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.247383    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"]
Oct 30 13:46:26.252808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:26.252785    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f97234-c597c8447-q8xqp"]
Oct 30 13:46:27.279983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:27.279948    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"]
Oct 30 13:46:27.280450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:27.280223    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy" containerID="cri-o://1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb" gracePeriod=30
Oct 30 13:46:28.076366 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:28.076322    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b6e91c-2f71-4503-b75c-4dba639ae8d7" path="/var/lib/kubelet/pods/f3b6e91c-2f71-4503-b75c-4dba639ae8d7/volumes"
Oct 30 13:46:30.189548 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:30.189513    4429 patch_prober.go:29] interesting pod/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:46:30.189969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:30.189583    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:46:32.559377 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.559310    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:46:32.707734 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.707703    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-data\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.707958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.707770    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-socket\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.707958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.707821    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-podinfo\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.707958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.707902    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8475e8-fba5-4fda-9538-1ae39efe931e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8475e8-fba5-4fda-9538-1ae39efe931e-istiod-ca-cert\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.707958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.707949    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-kube-api-access-ncvg6\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-kube-api-access-ncvg6\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.708189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.707988    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-data" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.708579 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.708548    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-envoy\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.708698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.708593    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-socket" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.708698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.708645    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-credential-socket\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.708698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.708655    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8475e8-fba5-4fda-9538-1ae39efe931e-istiod-ca-cert" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:32.708698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.708695    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-certs\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.708948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.708746    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-token\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-token\") pod \"5a8475e8-fba5-4fda-9538-1ae39efe931e\" (UID: \"5a8475e8-fba5-4fda-9538-1ae39efe931e\") "
Oct 30 13:46:32.708948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.708939    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-credential-socket" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.709139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.709114    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-certs" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.710169 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.710141    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.710264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.710174    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.710264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.710198    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.710264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.710212    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.710264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.710228    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8475e8-fba5-4fda-9538-1ae39efe931e-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.712730 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.712694    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-envoy" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:32.712851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.712764    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-token" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:32.712851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.712810    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-kube-api-access-ncvg6" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "kube-api-access-ncvg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:32.718676 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.718646    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-podinfo" pod "5a8475e8-fba5-4fda-9538-1ae39efe931e" (UID: "5a8475e8-fba5-4fda-9538-1ae39efe931e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:32.811690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.811580    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.811690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.811628    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvg6\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-kube-api-access-ncvg6\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.811690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.811641    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:32.811690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:32.811653    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8475e8-fba5-4fda-9538-1ae39efe931e-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:33.016001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.015964    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8" podUID="dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:46:33.241581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.241476    4429 generic.go:356] "Generic (PLEG): container finished" podID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerID="1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb" exitCode=0
Oct 30 13:46:33.241581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.241545    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"
Oct 30 13:46:33.241870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.241568    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" event={"ID":"5a8475e8-fba5-4fda-9538-1ae39efe931e","Type":"ContainerDied","Data":"1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb"}
Oct 30 13:46:33.241870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.241641    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs" event={"ID":"5a8475e8-fba5-4fda-9538-1ae39efe931e","Type":"ContainerDied","Data":"1bd83e6f5c63fc49bfd32b407671d8f5c871f41ba320a0d5eca1adb6249df53a"}
Oct 30 13:46:33.241870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.241672    4429 scope.go:122] "RemoveContainer" containerID="1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb"
Oct 30 13:46:33.253712 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.253691    4429 scope.go:122] "RemoveContainer" containerID="1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb"
Oct 30 13:46:33.254076 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:33.254050    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb\": container with ID starting with 1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb not found: ID does not exist" containerID="1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb"
Oct 30 13:46:33.254152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.254085    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb"} err="failed to get container status \"1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb\": rpc error: code = NotFound desc = could not find container \"1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb\": container with ID starting with 1b9441e78f325b03b7338537b55fe41f6aac00cb5c9fc22c3caa165cef3791cb not found: ID does not exist"
Oct 30 13:46:33.327706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.327670    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"]
Oct 30 13:46:33.339606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:33.339573    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7b7407-openshift-default-76bcb76d95-8dxfs"]
Oct 30 13:46:34.077506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:34.077467    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" path="/var/lib/kubelet/pods/5a8475e8-fba5-4fda-9538-1ae39efe931e/volumes"
Oct 30 13:46:37.684360 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:37.684327    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:46:37.798911 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:37.798874    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-plugins-config-volume\") pod \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\" (UID: \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\") "
Oct 30 13:46:37.799074 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:37.799006    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-kube-api-access-vbhgf\" (UniqueName: \"kubernetes.io/projected/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-kube-api-access-vbhgf\") pod \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\" (UID: \"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f\") "
Oct 30 13:46:37.799276 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:37.799252    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-plugins-config-volume" pod "dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" (UID: "dc0ecdf2-ad30-477b-8374-cdeeb746ec0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:37.799754 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:37.799736    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:37.801606 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:37.801582    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-kube-api-access-vbhgf" pod "dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" (UID: "dc0ecdf2-ad30-477b-8374-cdeeb746ec0f"). InnerVolumeSpecName "kube-api-access-vbhgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:37.901072 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:37.901032    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhgf\" (UniqueName: \"kubernetes.io/projected/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f-kube-api-access-vbhgf\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:38.297694 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.297657    4429 generic.go:356] "Generic (PLEG): container finished" podID="dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" containerID="a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c" exitCode=0
Oct 30 13:46:38.297920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.297712    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"
Oct 30 13:46:38.297920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.297738    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8" event={"ID":"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f","Type":"ContainerDied","Data":"a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c"}
Oct 30 13:46:38.297920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.297812    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8" event={"ID":"dc0ecdf2-ad30-477b-8374-cdeeb746ec0f","Type":"ContainerDied","Data":"2fccb27ea3fdcf60cbc7955e4003f859af6dcca2be9149ce814fe3bc33dad379"}
Oct 30 13:46:38.297920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.297843    4429 scope.go:122] "RemoveContainer" containerID="a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c"
Oct 30 13:46:38.309178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.309158    4429 scope.go:122] "RemoveContainer" containerID="a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c"
Oct 30 13:46:38.309543 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:38.309516    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c\": container with ID starting with a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c not found: ID does not exist" containerID="a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c"
Oct 30 13:46:38.309651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.309551    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c"} err="failed to get container status \"a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c\": rpc error: code = NotFound desc = could not find container \"a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c\": container with ID starting with a784dd142f5cb5cffd639c78f55e4e2ee9f43d5f1491669ba377f00fad37742c not found: ID does not exist"
Oct 30 13:46:38.358183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.358103    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"]
Oct 30 13:46:38.371620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:38.371587    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af8369-6b99d499cf-r6vw8"]
Oct 30 13:46:39.366795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.366241    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"]
Oct 30 13:46:39.366795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.366522    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk" podUID="015afa3f-7b83-4548-974f-f8f0aeb2c7ec" containerName="maas-api" containerID="cri-o://ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110" gracePeriod=30
Oct 30 13:46:39.695687 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.695650    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:46:39.833809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.833777    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-kube-api-access-tph6w\" (UniqueName: \"kubernetes.io/projected/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-kube-api-access-tph6w\") pod \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") "
Oct 30 13:46:39.834002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.833877    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-openshift-service-ca\") pod \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") "
Oct 30 13:46:39.834097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.834081    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-metrics-certs\") pod \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") "
Oct 30 13:46:39.834161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.834144    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls\") pod \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\" (UID: \"015afa3f-7b83-4548-974f-f8f0aeb2c7ec\") "
Oct 30 13:46:39.834378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.834345    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-openshift-service-ca" pod "015afa3f-7b83-4548-974f-f8f0aeb2c7ec" (UID: "015afa3f-7b83-4548-974f-f8f0aeb2c7ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:39.834921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.834895    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:39.836321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.836267    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-metrics-certs" pod "015afa3f-7b83-4548-974f-f8f0aeb2c7ec" (UID: "015afa3f-7b83-4548-974f-f8f0aeb2c7ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:39.836409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.836331    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-kube-api-access-tph6w" pod "015afa3f-7b83-4548-974f-f8f0aeb2c7ec" (UID: "015afa3f-7b83-4548-974f-f8f0aeb2c7ec"). InnerVolumeSpecName "kube-api-access-tph6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:39.836479 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.836464    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls" pod "015afa3f-7b83-4548-974f-f8f0aeb2c7ec" (UID: "015afa3f-7b83-4548-974f-f8f0aeb2c7ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:39.936645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.936616    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:39.936645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.936649    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:39.936856 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:39.936658    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tph6w\" (UniqueName: \"kubernetes.io/projected/015afa3f-7b83-4548-974f-f8f0aeb2c7ec-kube-api-access-tph6w\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:40.077948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.077913    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" path="/var/lib/kubelet/pods/dc0ecdf2-ad30-477b-8374-cdeeb746ec0f/volumes"
Oct 30 13:46:40.316853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.316728    4429 generic.go:356] "Generic (PLEG): container finished" podID="015afa3f-7b83-4548-974f-f8f0aeb2c7ec" containerID="ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110" exitCode=0
Oct 30 13:46:40.317080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.316942    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk" event={"ID":"015afa3f-7b83-4548-974f-f8f0aeb2c7ec","Type":"ContainerDied","Data":"ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110"}
Oct 30 13:46:40.317080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.317004    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk" event={"ID":"015afa3f-7b83-4548-974f-f8f0aeb2c7ec","Type":"ContainerDied","Data":"47e633aee476f3dc1da71972542e61c15d98cdf540c3432aeff33fff2ea6bab8"}
Oct 30 13:46:40.317080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.317035    4429 scope.go:122] "RemoveContainer" containerID="ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110"
Oct 30 13:46:40.317304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.317173    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"
Oct 30 13:46:40.328404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.328382    4429 scope.go:122] "RemoveContainer" containerID="ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110"
Oct 30 13:46:40.328675 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:40.328654    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110\": container with ID starting with ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110 not found: ID does not exist" containerID="ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110"
Oct 30 13:46:40.328722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.328684    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110"} err="failed to get container status \"ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110\": rpc error: code = NotFound desc = could not find container \"ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110\": container with ID starting with ae12c18a0b117eee104bcbaab5a375931d0838241b04507bec4a08b5e37cf110 not found: ID does not exist"
Oct 30 13:46:40.377705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.377669    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"]
Oct 30 13:46:40.391599 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:40.391568    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9a93bd-7b59f94874-sg6tk"]
Oct 30 13:46:42.080636 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:42.080591    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015afa3f-7b83-4548-974f-f8f0aeb2c7ec" path="/var/lib/kubelet/pods/015afa3f-7b83-4548-974f-f8f0aeb2c7ec/volumes"
Oct 30 13:46:42.373395 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:42.373279    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:46:48.999010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:48.998968    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"]
Oct 30 13:46:48.999878 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:48.999211    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5" podUID="fc20bc63-5f53-491a-9efe-b3f5f02e59dd" containerName="maas-api" containerID="cri-o://a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9" gracePeriod=30
Oct 30 13:46:49.108205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.108169    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"]
Oct 30 13:46:49.110000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.109946    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy" containerID="cri-o://9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51" gracePeriod=30
Oct 30 13:46:49.358100 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.358071    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:46:49.386660 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.386627    4429 generic.go:356] "Generic (PLEG): container finished" podID="fc20bc63-5f53-491a-9efe-b3f5f02e59dd" containerID="a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9" exitCode=0
Oct 30 13:46:49.386830 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.386748    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"
Oct 30 13:46:49.386905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.386752    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5" event={"ID":"fc20bc63-5f53-491a-9efe-b3f5f02e59dd","Type":"ContainerDied","Data":"a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9"}
Oct 30 13:46:49.386905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.386878    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5" event={"ID":"fc20bc63-5f53-491a-9efe-b3f5f02e59dd","Type":"ContainerDied","Data":"cd77cd04a058e7e46c67d5639bb7057e33389539f3f326da115147257c2a70c5"}
Oct 30 13:46:49.386988 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.386911    4429 scope.go:122] "RemoveContainer" containerID="a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9"
Oct 30 13:46:49.400177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.400155    4429 scope.go:122] "RemoveContainer" containerID="a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9"
Oct 30 13:46:49.400592 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:49.400561    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9\": container with ID starting with a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9 not found: ID does not exist" containerID="a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9"
Oct 30 13:46:49.400706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.400602    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9"} err="failed to get container status \"a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9\": rpc error: code = NotFound desc = could not find container \"a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9\": container with ID starting with a6b3486814e5fba3faea8b80ea9df4701a4d1e864005f207c5d09e34d6eaa3a9 not found: ID does not exist"
Oct 30 13:46:49.513478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.509604    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-maas-api-tls\") pod \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") "
Oct 30 13:46:49.513478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.509705    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-metrics-certs\") pod \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") "
Oct 30 13:46:49.513478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.509823    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-openshift-service-ca\") pod \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") "
Oct 30 13:46:49.513478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.509968    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-kube-api-access-l8cdf\" (UniqueName: \"kubernetes.io/projected/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-kube-api-access-l8cdf\") pod \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\" (UID: \"fc20bc63-5f53-491a-9efe-b3f5f02e59dd\") "
Oct 30 13:46:49.513478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.511669    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-openshift-service-ca" pod "fc20bc63-5f53-491a-9efe-b3f5f02e59dd" (UID: "fc20bc63-5f53-491a-9efe-b3f5f02e59dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:49.517525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.517498    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-metrics-certs" pod "fc20bc63-5f53-491a-9efe-b3f5f02e59dd" (UID: "fc20bc63-5f53-491a-9efe-b3f5f02e59dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:49.520246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.520214    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-kube-api-access-l8cdf" pod "fc20bc63-5f53-491a-9efe-b3f5f02e59dd" (UID: "fc20bc63-5f53-491a-9efe-b3f5f02e59dd"). InnerVolumeSpecName "kube-api-access-l8cdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:49.520652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.520629    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-maas-api-tls" pod "fc20bc63-5f53-491a-9efe-b3f5f02e59dd" (UID: "fc20bc63-5f53-491a-9efe-b3f5f02e59dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:49.612514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.612478    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8cdf\" (UniqueName: \"kubernetes.io/projected/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-kube-api-access-l8cdf\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:49.612514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.612506    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:49.612514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.612516    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:49.612514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.612524    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc20bc63-5f53-491a-9efe-b3f5f02e59dd-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:49.748932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.748899    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"]
Oct 30 13:46:49.752524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:49.752498    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8369-6545fbd4f7-p97q5"]
Oct 30 13:46:50.077158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:50.077122    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc20bc63-5f53-491a-9efe-b3f5f02e59dd" path="/var/lib/kubelet/pods/fc20bc63-5f53-491a-9efe-b3f5f02e59dd/volumes"
Oct 30 13:46:53.559380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:53.559337    4429 patch_prober.go:29] interesting pod/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:46:53.559764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:53.559415    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:46:54.402359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.402263    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:46:54.425940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.425900    4429 generic.go:356] "Generic (PLEG): container finished" podID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerID="9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51" exitCode=0
Oct 30 13:46:54.426119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.425986    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"
Oct 30 13:46:54.426119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.425984    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" event={"ID":"31b4e94b-6359-4b53-8bbb-501c9cd698ce","Type":"ContainerDied","Data":"9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51"}
Oct 30 13:46:54.426119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.426049    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm" event={"ID":"31b4e94b-6359-4b53-8bbb-501c9cd698ce","Type":"ContainerDied","Data":"8d2a00e9fe8b36e7e90d9a7b2306bbfe864327947263266bbf4d9a52d9ec861f"}
Oct 30 13:46:54.426119 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.426079    4429 scope.go:122] "RemoveContainer" containerID="9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51"
Oct 30 13:46:54.444865 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.442822    4429 scope.go:122] "RemoveContainer" containerID="9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51"
Oct 30 13:46:54.444937 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:46:54.444881    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51\": container with ID starting with 9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51 not found: ID does not exist" containerID="9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51"
Oct 30 13:46:54.444937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.444917    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51"} err="failed to get container status \"9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51\": rpc error: code = NotFound desc = could not find container \"9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51\": container with ID starting with 9dac104f1f84b01ce6bf76fc0f8f363711a2e85859acc170f6e78d889af22e51 not found: ID does not exist"
Oct 30 13:46:54.484626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.484588    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-envoy\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.484850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.484688    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-socket\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.484850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.484742    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-token\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-token\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.484850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.484784    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istiod-ca-cert\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.484850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.484822    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-certs\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.485172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.484907    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-kube-api-access-2pwnf\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-kube-api-access-2pwnf\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.485889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.485839    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-credential-socket\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.485889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.485872    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-certs" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.486108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.486045    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-podinfo\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.486108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.486099    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-data\") pod \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\" (UID: \"31b4e94b-6359-4b53-8bbb-501c9cd698ce\") "
Oct 30 13:46:54.486211 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.486171    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istiod-ca-cert" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:54.491736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.486863    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.491736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.486891    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.491736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.487081    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-data" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.491736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.487201    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-socket" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.491736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.487667    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-credential-socket" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.492226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.492182    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-token" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:54.492354 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.492243    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-kube-api-access-2pwnf" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "kube-api-access-2pwnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:54.492618 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.492581    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-podinfo" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:46:54.492751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.492645    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-envoy" pod "31b4e94b-6359-4b53-8bbb-501c9cd698ce" (UID: "31b4e94b-6359-4b53-8bbb-501c9cd698ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:46:54.588052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.588014    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.588052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.588049    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.588052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.588059    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pwnf\" (UniqueName: \"kubernetes.io/projected/31b4e94b-6359-4b53-8bbb-501c9cd698ce-kube-api-access-2pwnf\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.588596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.588069    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.588596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.588078    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.588596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.588086    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.588596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.588094    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b4e94b-6359-4b53-8bbb-501c9cd698ce-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.798379 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.798338    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"]
Oct 30 13:46:54.811902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:54.811863    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f97234-openshift-default-f5dd5c7d7-7bfvm"]
Oct 30 13:46:56.078019 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.077925    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" path="/var/lib/kubelet/pods/31b4e94b-6359-4b53-8bbb-501c9cd698ce/volumes"
Oct 30 13:46:56.461002 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.460931    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"]
Oct 30 13:46:56.462158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462127    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:46:56.462158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462154    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:46:56.462158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462171    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:46:56.462158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462179    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:46:56.462158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462202    4429 cpu_manager.go:414] "containerMap: removing container" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.462158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462210    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462220    4429 cpu_manager.go:414] "containerMap: removing container" podUID="641780cb-2978-4633-af57-2463feb3acef" containerName="revoke-keys"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462228    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641780cb-2978-4633-af57-2463feb3acef" containerName="revoke-keys"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462250    4429 cpu_manager.go:414] "containerMap: removing container" podUID="8cf78b02-2d46-480c-9b35-0511616757cf" containerName="payload-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462258    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf78b02-2d46-480c-9b35-0511616757cf" containerName="payload-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462269    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462276    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462310    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0af1c0aa-06d3-401e-94ce-021effe0ba5f" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462317    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af1c0aa-06d3-401e-94ce-021effe0ba5f" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462327    4429 cpu_manager.go:414] "containerMap: removing container" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462334    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462349    4429 cpu_manager.go:414] "containerMap: removing container" podUID="fc20bc63-5f53-491a-9efe-b3f5f02e59dd" containerName="maas-api"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462356    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc20bc63-5f53-491a-9efe-b3f5f02e59dd" containerName="maas-api"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462366    4429 cpu_manager.go:414] "containerMap: removing container" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462373    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462398    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462406    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462420    4429 cpu_manager.go:414] "containerMap: removing container" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462428    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462442    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462449    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462463    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b8dcaf96-99c3-42c7-8e03-4f45e4433287" containerName="payload-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462469    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8dcaf96-99c3-42c7-8e03-4f45e4433287" containerName="payload-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462481    4429 cpu_manager.go:414] "containerMap: removing container" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462488    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462504    4429 cpu_manager.go:414] "containerMap: removing container" podUID="4e4761cc-99df-4415-9b05-153a79dfc5fc" containerName="payload-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462511    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4761cc-99df-4415-9b05-153a79dfc5fc" containerName="payload-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462524    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462531    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462547    4429 cpu_manager.go:414] "containerMap: removing container" podUID="f3b6e91c-2f71-4503-b75c-4dba639ae8d7" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462555    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b6e91c-2f71-4503-b75c-4dba639ae8d7" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462580    4429 cpu_manager.go:414] "containerMap: removing container" podUID="efa69d05-7642-40a4-8a81-ae56b7952142" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462587    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa69d05-7642-40a4-8a81-ae56b7952142" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462598    4429 cpu_manager.go:414] "containerMap: removing container" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462607    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462616    4429 cpu_manager.go:414] "containerMap: removing container" podUID="dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462623    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462634    4429 cpu_manager.go:414] "containerMap: removing container" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462640    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462653    4429 cpu_manager.go:414] "containerMap: removing container" podUID="015afa3f-7b83-4548-974f-f8f0aeb2c7ec" containerName="maas-api"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462660    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015afa3f-7b83-4548-974f-f8f0aeb2c7ec" containerName="maas-api"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462675    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1e68bd2b-8c5d-462a-a91d-c6ba92965718" containerName="payload-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462682    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e68bd2b-8c5d-462a-a91d-c6ba92965718" containerName="payload-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462695    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c241fb9d-2a90-474e-be70-2abca4252e27" containerName="maas-api"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462702    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c241fb9d-2a90-474e-be70-2abca4252e27" containerName="maas-api"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462715    4429 cpu_manager.go:414] "containerMap: removing container" podUID="719a6fc2-d535-4266-b8c7-52c20254c55f" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462722    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719a6fc2-d535-4266-b8c7-52c20254c55f" containerName="payload-pre-processing"
Oct 30 13:46:56.462706 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462737    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462744    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462758    4429 cpu_manager.go:414] "containerMap: removing container" podUID="2c64c968-129c-4f4e-806e-a01a1a59e267" containerName="payload-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462765    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c64c968-129c-4f4e-806e-a01a1a59e267" containerName="payload-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462776    4429 cpu_manager.go:414] "containerMap: removing container" podUID="35af2313-f69b-4be2-ba2e-eb9987eeae3b" containerName="maas-api"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462783    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35af2313-f69b-4be2-ba2e-eb9987eeae3b" containerName="maas-api"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462960    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462971    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8475e8-fba5-4fda-9538-1ae39efe931e" containerName="istio-proxy"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462982    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462990    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8dcaf96-99c3-42c7-8e03-4f45e4433287" containerName="payload-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.462999    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="920d1596-271c-4f07-bd46-061f9adc053d" containerName="istio-proxy"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463010    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4761cc-99df-4415-9b05-153a79dfc5fc" containerName="payload-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463019    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463031    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b6e91c-2f71-4503-b75c-4dba639ae8d7" containerName="payload-pre-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463039    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463049    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa69d05-7642-40a4-8a81-ae56b7952142" containerName="payload-pre-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463059    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463068    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0ecdf2-ad30-477b-8374-cdeeb746ec0f" containerName="payload-pre-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463075    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463084    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="015afa3f-7b83-4548-974f-f8f0aeb2c7ec" containerName="maas-api"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463093    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e68bd2b-8c5d-462a-a91d-c6ba92965718" containerName="payload-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463102    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c241fb9d-2a90-474e-be70-2abca4252e27" containerName="maas-api"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463110    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="719a6fc2-d535-4266-b8c7-52c20254c55f" containerName="payload-pre-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463119    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce57651f-cec8-4d1b-b46d-04b081bd0ac8" containerName="istio-proxy"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463128    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c64c968-129c-4f4e-806e-a01a1a59e267" containerName="payload-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463136    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="35af2313-f69b-4be2-ba2e-eb9987eeae3b" containerName="maas-api"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463143    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463152    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463162    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463170    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="641780cb-2978-4633-af57-2463feb3acef" containerName="revoke-keys"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463180    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf78b02-2d46-480c-9b35-0511616757cf" containerName="payload-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463191    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af1c0aa-06d3-401e-94ce-021effe0ba5f" containerName="payload-pre-processing"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463199    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b4e94b-6359-4b53-8bbb-501c9cd698ce" containerName="istio-proxy"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463209    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc20bc63-5f53-491a-9efe-b3f5f02e59dd" containerName="maas-api"
Oct 30 13:46:56.466489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.463217    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="254fca5d-4794-489f-bb87-fedcd96d2e70" containerName="cleanup"
Oct 30 13:46:56.472124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.472093    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:56.505584 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.505406    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d01cd5a5\""
Oct 30 13:46:56.505584 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.505530    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d01cd5a5-dockercfg-n62dw\""
Oct 30 13:46:56.537605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.537235    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x7m9\" (UniqueName: \"kubernetes.io/projected/c3b857b2-4411-480a-a755-699c535ccb25-kube-api-access-9x7m9\") pod \"payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg\" (UID: \"c3b857b2-4411-480a-a755-699c535ccb25\") " pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:56.537605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.537394    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b857b2-4411-480a-a755-699c535ccb25-plugins-config-volume\") pod \"payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg\" (UID: \"c3b857b2-4411-480a-a755-699c535ccb25\") " pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:56.554323 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.552537    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"]
Oct 30 13:46:56.638943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.638907    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x7m9\" (UniqueName: \"kubernetes.io/projected/c3b857b2-4411-480a-a755-699c535ccb25-kube-api-access-9x7m9\") pod \"payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg\" (UID: \"c3b857b2-4411-480a-a755-699c535ccb25\") " pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:56.639145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.638990    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b857b2-4411-480a-a755-699c535ccb25-plugins-config-volume\") pod \"payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg\" (UID: \"c3b857b2-4411-480a-a755-699c535ccb25\") " pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:56.639779 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.639758    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b857b2-4411-480a-a755-699c535ccb25-plugins-config-volume\") pod \"payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg\" (UID: \"c3b857b2-4411-480a-a755-699c535ccb25\") " pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:56.665494 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.665448    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x7m9\" (UniqueName: \"kubernetes.io/projected/c3b857b2-4411-480a-a755-699c535ccb25-kube-api-access-9x7m9\") pod \"payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg\" (UID: \"c3b857b2-4411-480a-a755-699c535ccb25\") " pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:56.854938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:56.854853    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:57.164525 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:46:57.164492    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b857b2_4411_480a_a755_699c535ccb25.slice/crio-b4ba126fbe82ec31ac8d2e3252b29a2f9d97f23db5fbc6afa65d1d847fa84b53 WatchSource:0}: Error finding container b4ba126fbe82ec31ac8d2e3252b29a2f9d97f23db5fbc6afa65d1d847fa84b53: Status 404 returned error can't find the container with id b4ba126fbe82ec31ac8d2e3252b29a2f9d97f23db5fbc6afa65d1d847fa84b53
Oct 30 13:46:57.165529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:57.165501    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"]
Oct 30 13:46:57.456894 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:57.456853    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg" event={"ID":"c3b857b2-4411-480a-a755-699c535ccb25","Type":"ContainerStarted","Data":"c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480"}
Oct 30 13:46:57.457097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:57.456921    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:46:57.457097 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:57.456942    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg" event={"ID":"c3b857b2-4411-480a-a755-699c535ccb25","Type":"ContainerStarted","Data":"b4ba126fbe82ec31ac8d2e3252b29a2f9d97f23db5fbc6afa65d1d847fa84b53"}
Oct 30 13:46:57.504891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:46:57.504832    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg" podStartSLOduration=1.504818043 podStartE2EDuration="1.504818043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:57.503524776 +0000 UTC m=+4475.559018601" watchObservedRunningTime="2026-10-30 13:46:57.504818043 +0000 UTC m=+4475.560311866"
Oct 30 13:47:08.468303 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:08.468261    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:47:12.373690 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:47:12.373649    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:47:40.040083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.040047    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"]
Oct 30 13:47:40.041866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.041830    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:47:40.041866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.041856    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:47:40.042028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.041923    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:47:40.042028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.041932    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0fadc2-ca92-476c-8c21-81f946cfbe21" containerName="cleanup"
Oct 30 13:47:40.042109 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.042095    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bad93c-a0e2-4807-821b-08f59773174f" containerName="cleanup"
Oct 30 13:47:40.042157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.042117    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e22b00a-2cf6-4549-92ba-fa7019138663" containerName="cleanup"
Oct 30 13:47:40.048397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.048371    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.137734 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.119053    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d01cd5a5-serving-cert\""
Oct 30 13:47:40.342445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.240306    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xbj\" (UniqueName: \"kubernetes.io/projected/d5daadbb-275f-4346-820c-d9bbb818202c-kube-api-access-82xbj\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.342445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.240452    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-metrics-certs\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.342445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.240639    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-maas-api-tls\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.342445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.240699    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5daadbb-275f-4346-820c-d9bbb818202c-openshift-service-ca\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.342848 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.342502    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-maas-api-tls\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.342848 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.342705    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5daadbb-275f-4346-820c-d9bbb818202c-openshift-service-ca\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.342848 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.342811    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xbj\" (UniqueName: \"kubernetes.io/projected/d5daadbb-275f-4346-820c-d9bbb818202c-kube-api-access-82xbj\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.343035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.342875    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-metrics-certs\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.343601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.343575    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5daadbb-275f-4346-820c-d9bbb818202c-openshift-service-ca\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.346059 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.346030    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-maas-api-tls\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.346775 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.346755    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-metrics-certs\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.346931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.346844    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"]
Oct 30 13:47:40.371744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.371677    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xbj\" (UniqueName: \"kubernetes.io/projected/d5daadbb-275f-4346-820c-d9bbb818202c-kube-api-access-82xbj\") pod \"maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.564105 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.564075    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:40.892009 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.891975    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"]
Oct 30 13:47:40.892318 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:47:40.892272    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5daadbb_275f_4346_820c_d9bbb818202c.slice/crio-53c07ef92c69cf2b8bfd456ce6dbe1b10c4f7834b8ccffcbc7c469c451ad8994 WatchSource:0}: Error finding container 53c07ef92c69cf2b8bfd456ce6dbe1b10c4f7834b8ccffcbc7c469c451ad8994: Status 404 returned error can't find the container with id 53c07ef92c69cf2b8bfd456ce6dbe1b10c4f7834b8ccffcbc7c469c451ad8994
Oct 30 13:47:40.893756 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:40.893739    4429 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:47:41.850597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:41.850515    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx" event={"ID":"d5daadbb-275f-4346-820c-d9bbb818202c","Type":"ContainerStarted","Data":"59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181"}
Oct 30 13:47:41.850597 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:41.850575    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx" event={"ID":"d5daadbb-275f-4346-820c-d9bbb818202c","Type":"ContainerStarted","Data":"53c07ef92c69cf2b8bfd456ce6dbe1b10c4f7834b8ccffcbc7c469c451ad8994"}
Oct 30 13:47:41.851028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:41.850614    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:47:41.902044 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:41.901994    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx" podStartSLOduration=1.354504189 podStartE2EDuration="1.901979769s" totalImagesPullingTime="547.47558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:41.9014975 +0000 UTC m=+4519.956991323" watchObservedRunningTime="2026-10-30 13:47:41.901979769 +0000 UTC m=+4519.957473593"
Oct 30 13:47:42.373176 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:47:42.373140    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:47:47.862605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:47:47.862577    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:48:12.375250 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:48:12.375159    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:48:42.376043 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:48:42.376004    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:49:12.373490 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:49:12.373448    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:49:42.374883 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:49:42.374804    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:50:00.711945 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:00.711909    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"]
Oct 30 13:50:00.712454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:00.712130    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9" podUID="4a6244f0-a709-46d6-bafd-dbad9edb2d74" containerName="maas-api" containerID="cri-o://f917b683a59a8ef507f75033dcbe2c521b0a9ba2d40e610e833768873e18694e" gracePeriod=30
Oct 30 13:50:00.939314 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:00.939250    4429 generic.go:356] "Generic (PLEG): container finished" podID="4a6244f0-a709-46d6-bafd-dbad9edb2d74" containerID="f917b683a59a8ef507f75033dcbe2c521b0a9ba2d40e610e833768873e18694e" exitCode=0
Oct 30 13:50:00.939512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:00.939336    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9" event={"ID":"4a6244f0-a709-46d6-bafd-dbad9edb2d74","Type":"ContainerDied","Data":"f917b683a59a8ef507f75033dcbe2c521b0a9ba2d40e610e833768873e18694e"}
Oct 30 13:50:01.046709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.046683    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:50:01.233976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.233943    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6244f0-a709-46d6-bafd-dbad9edb2d74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6244f0-a709-46d6-bafd-dbad9edb2d74-openshift-service-ca\") pod \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") "
Oct 30 13:50:01.233976 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.233979    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6244f0-a709-46d6-bafd-dbad9edb2d74-kube-api-access-4rv7d\" (UniqueName: \"kubernetes.io/projected/4a6244f0-a709-46d6-bafd-dbad9edb2d74-kube-api-access-4rv7d\") pod \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") "
Oct 30 13:50:01.234255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.234020    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-metrics-certs\") pod \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") "
Oct 30 13:50:01.234255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.234137    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-maas-api-tls\") pod \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\" (UID: \"4a6244f0-a709-46d6-bafd-dbad9edb2d74\") "
Oct 30 13:50:01.234429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.234402    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6244f0-a709-46d6-bafd-dbad9edb2d74-openshift-service-ca" pod "4a6244f0-a709-46d6-bafd-dbad9edb2d74" (UID: "4a6244f0-a709-46d6-bafd-dbad9edb2d74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:01.234786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.234758    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6244f0-a709-46d6-bafd-dbad9edb2d74-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:01.236850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.236824    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6244f0-a709-46d6-bafd-dbad9edb2d74-kube-api-access-4rv7d" pod "4a6244f0-a709-46d6-bafd-dbad9edb2d74" (UID: "4a6244f0-a709-46d6-bafd-dbad9edb2d74"). InnerVolumeSpecName "kube-api-access-4rv7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:01.237741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.237721    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-maas-api-tls" pod "4a6244f0-a709-46d6-bafd-dbad9edb2d74" (UID: "4a6244f0-a709-46d6-bafd-dbad9edb2d74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:01.237996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.237973    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-metrics-certs" pod "4a6244f0-a709-46d6-bafd-dbad9edb2d74" (UID: "4a6244f0-a709-46d6-bafd-dbad9edb2d74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:01.336186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.336102    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:01.336186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.336137    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rv7d\" (UniqueName: \"kubernetes.io/projected/4a6244f0-a709-46d6-bafd-dbad9edb2d74-kube-api-access-4rv7d\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:01.336186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.336151    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a6244f0-a709-46d6-bafd-dbad9edb2d74-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:01.948026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.947982    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9" event={"ID":"4a6244f0-a709-46d6-bafd-dbad9edb2d74","Type":"ContainerDied","Data":"3f589bbbd151d8572058f137f4ea36b388b52fbc4f3d2e66b4eac12a67784969"}
Oct 30 13:50:01.948026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.948007    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"
Oct 30 13:50:01.948601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:01.948044    4429 scope.go:122] "RemoveContainer" containerID="f917b683a59a8ef507f75033dcbe2c521b0a9ba2d40e610e833768873e18694e"
Oct 30 13:50:02.027158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:02.027129    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"]
Oct 30 13:50:02.039468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:02.039444    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0505b0-7f4fd8f545-jqqk9"]
Oct 30 13:50:02.077518 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:02.077494    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6244f0-a709-46d6-bafd-dbad9edb2d74" path="/var/lib/kubelet/pods/4a6244f0-a709-46d6-bafd-dbad9edb2d74/volumes"
Oct 30 13:50:10.369909 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:10.369871    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"]
Oct 30 13:50:10.370359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:10.370090    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy" containerID="cri-o://8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d" gracePeriod=30
Oct 30 13:50:11.050176 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:11.050106    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"]
Oct 30 13:50:11.055975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:11.055938    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0855dd6-9r8r7"]
Oct 30 13:50:11.754413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:11.754375    4429 patch_prober.go:29] interesting pod/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:50:11.754890 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:11.754449    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:50:12.077218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:12.077135    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641780cb-2978-4633-af57-2463feb3acef" path="/var/lib/kubelet/pods/641780cb-2978-4633-af57-2463feb3acef/volumes"
Oct 30 13:50:12.373496 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:50:12.373416    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:50:15.670689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.670662    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:50:15.736045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736015    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-kube-api-access-j98mk\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-kube-api-access-j98mk\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.736258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736073    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-envoy\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.736258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736187    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f924a4-fb89-4052-bf95-790c1129da55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f924a4-fb89-4052-bf95-790c1129da55-istiod-ca-cert\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.736258 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736249    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-data\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.736512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736315    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-istio-token\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-istio-token\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.736512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736353    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-socket\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.736512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736387    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-credential-socket\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.736512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736426    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-certs\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.736512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.736488    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3f924a4-fb89-4052-bf95-790c1129da55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f924a4-fb89-4052-bf95-790c1129da55-istio-podinfo\") pod \"d3f924a4-fb89-4052-bf95-790c1129da55\" (UID: \"d3f924a4-fb89-4052-bf95-790c1129da55\") "
Oct 30 13:50:15.738337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.737938    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-credential-socket" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:50:15.738337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.738135    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-socket" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:50:15.738550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.738414    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-data" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:50:15.738621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.738587    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-certs" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:50:15.738866 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.738834    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f924a4-fb89-4052-bf95-790c1129da55-istiod-ca-cert" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:15.740118 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.740089    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-istio-token" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:15.740705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.740679    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-kube-api-access-j98mk" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "kube-api-access-j98mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:15.741551 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.741515    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3f924a4-fb89-4052-bf95-790c1129da55-istio-podinfo" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:50:15.741775 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.741750    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-envoy" pod "d3f924a4-fb89-4052-bf95-790c1129da55" (UID: "d3f924a4-fb89-4052-bf95-790c1129da55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:50:15.839043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839007    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:15.839043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839037    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:15.839043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839046    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:15.839347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839056    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:15.839347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839067    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:15.839347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839077    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3f924a4-fb89-4052-bf95-790c1129da55-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:15.839347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839104    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j98mk\" (UniqueName: \"kubernetes.io/projected/d3f924a4-fb89-4052-bf95-790c1129da55-kube-api-access-j98mk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:15.839347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839113    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3f924a4-fb89-4052-bf95-790c1129da55-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:15.839347 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:15.839121    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3f924a4-fb89-4052-bf95-790c1129da55-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:16.055640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.055600    4429 generic.go:356] "Generic (PLEG): container finished" podID="d3f924a4-fb89-4052-bf95-790c1129da55" containerID="8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d" exitCode=0
Oct 30 13:50:16.055852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.055647    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" event={"ID":"d3f924a4-fb89-4052-bf95-790c1129da55","Type":"ContainerDied","Data":"8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d"}
Oct 30 13:50:16.055852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.055671    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"
Oct 30 13:50:16.055852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.055733    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb" event={"ID":"d3f924a4-fb89-4052-bf95-790c1129da55","Type":"ContainerDied","Data":"aefed12d19ae42e53879c884b532d53a49ee0617205f2f7b13d13ca79335d327"}
Oct 30 13:50:16.055852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.055764    4429 scope.go:122] "RemoveContainer" containerID="8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d"
Oct 30 13:50:16.068088 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.068057    4429 scope.go:122] "RemoveContainer" containerID="8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560"
Oct 30 13:50:16.082217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.082191    4429 scope.go:122] "RemoveContainer" containerID="8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d"
Oct 30 13:50:16.082572 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:50:16.082553    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d\": container with ID starting with 8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d not found: ID does not exist" containerID="8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d"
Oct 30 13:50:16.082643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.082583    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d"} err="failed to get container status \"8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d\": rpc error: code = NotFound desc = could not find container \"8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d\": container with ID starting with 8d58a83be4f9901b726157b8e9da8e861e1f7803e5484faeb20f260fa850917d not found: ID does not exist"
Oct 30 13:50:16.082643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.082602    4429 scope.go:122] "RemoveContainer" containerID="8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560"
Oct 30 13:50:16.082863 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:50:16.082843    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560\": container with ID starting with 8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560 not found: ID does not exist" containerID="8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560"
Oct 30 13:50:16.082919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.082872    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560"} err="failed to get container status \"8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560\": rpc error: code = NotFound desc = could not find container \"8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560\": container with ID starting with 8cf16a48045a651d0caeade0a33dbc04370aea136d04971448e01d97baf7b560 not found: ID does not exist"
Oct 30 13:50:16.116329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.116272    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"]
Oct 30 13:50:16.130587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:16.130548    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0505b0-openshift-default-d499f445-ks6mb"]
Oct 30 13:50:18.078861 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:18.078829    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" path="/var/lib/kubelet/pods/d3f924a4-fb89-4052-bf95-790c1129da55/volumes"
Oct 30 13:50:19.517691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:19.517655    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"]
Oct 30 13:50:21.418529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:21.418490    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"]
Oct 30 13:50:24.518246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:24.518173    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg" podUID="c3b857b2-4411-480a-a755-699c535ccb25" containerName="payload-processing" containerID="cri-o://c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480" gracePeriod=25
Oct 30 13:50:24.819104 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:24.819069    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:50:24.971062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:24.971020    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b857b2-4411-480a-a755-699c535ccb25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b857b2-4411-480a-a755-699c535ccb25-plugins-config-volume\") pod \"c3b857b2-4411-480a-a755-699c535ccb25\" (UID: \"c3b857b2-4411-480a-a755-699c535ccb25\") "
Oct 30 13:50:24.971487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:24.971467    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b857b2-4411-480a-a755-699c535ccb25-kube-api-access-9x7m9\" (UniqueName: \"kubernetes.io/projected/c3b857b2-4411-480a-a755-699c535ccb25-kube-api-access-9x7m9\") pod \"c3b857b2-4411-480a-a755-699c535ccb25\" (UID: \"c3b857b2-4411-480a-a755-699c535ccb25\") "
Oct 30 13:50:24.971570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:24.971517    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b857b2-4411-480a-a755-699c535ccb25-plugins-config-volume" pod "c3b857b2-4411-480a-a755-699c535ccb25" (UID: "c3b857b2-4411-480a-a755-699c535ccb25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:24.972012 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:24.971983    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b857b2-4411-480a-a755-699c535ccb25-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:24.974113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:24.974084    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b857b2-4411-480a-a755-699c535ccb25-kube-api-access-9x7m9" pod "c3b857b2-4411-480a-a755-699c535ccb25" (UID: "c3b857b2-4411-480a-a755-699c535ccb25"). InnerVolumeSpecName "kube-api-access-9x7m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:25.073506 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.073474    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x7m9\" (UniqueName: \"kubernetes.io/projected/c3b857b2-4411-480a-a755-699c535ccb25-kube-api-access-9x7m9\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:25.132919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.132883    4429 generic.go:356] "Generic (PLEG): container finished" podID="c3b857b2-4411-480a-a755-699c535ccb25" containerID="c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480" exitCode=0
Oct 30 13:50:25.133131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.132964    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg" event={"ID":"c3b857b2-4411-480a-a755-699c535ccb25","Type":"ContainerDied","Data":"c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480"}
Oct 30 13:50:25.133131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.133018    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"
Oct 30 13:50:25.133131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.133040    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg" event={"ID":"c3b857b2-4411-480a-a755-699c535ccb25","Type":"ContainerDied","Data":"b4ba126fbe82ec31ac8d2e3252b29a2f9d97f23db5fbc6afa65d1d847fa84b53"}
Oct 30 13:50:25.133131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.133071    4429 scope.go:122] "RemoveContainer" containerID="c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480"
Oct 30 13:50:25.145539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.145507    4429 scope.go:122] "RemoveContainer" containerID="c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480"
Oct 30 13:50:25.145955 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:50:25.145928    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480\": container with ID starting with c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480 not found: ID does not exist" containerID="c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480"
Oct 30 13:50:25.146000 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.145967    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480"} err="failed to get container status \"c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480\": rpc error: code = NotFound desc = could not find container \"c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480\": container with ID starting with c2db19bf88657ffd3d1756b465a1911a94f4aaba9cbdf99c73725afa812dd480 not found: ID does not exist"
Oct 30 13:50:25.190313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.190262    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"]
Oct 30 13:50:25.203826 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:25.203801    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d01cd5a5-8c5795c58-cgmdg"]
Oct 30 13:50:26.076842 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:26.076795    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b857b2-4411-480a-a755-699c535ccb25" path="/var/lib/kubelet/pods/c3b857b2-4411-480a-a755-699c535ccb25/volumes"
Oct 30 13:50:26.419386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:26.419230    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w" podUID="7f992cba-9c34-45a5-abc1-bfa307a8badb" containerName="payload-processing" containerID="cri-o://3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058" gracePeriod=25
Oct 30 13:50:27.744306 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.744251    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"]
Oct 30 13:50:27.745496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745469    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy"
Oct 30 13:50:27.745496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745497    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy"
Oct 30 13:50:27.745731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745594    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c3b857b2-4411-480a-a755-699c535ccb25" containerName="payload-processing"
Oct 30 13:50:27.745731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745603    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b857b2-4411-480a-a755-699c535ccb25" containerName="payload-processing"
Oct 30 13:50:27.745731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745614    4429 cpu_manager.go:414] "containerMap: removing container" podUID="4a6244f0-a709-46d6-bafd-dbad9edb2d74" containerName="maas-api"
Oct 30 13:50:27.745731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745621    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6244f0-a709-46d6-bafd-dbad9edb2d74" containerName="maas-api"
Oct 30 13:50:27.745960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745804    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy"
Oct 30 13:50:27.745960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745825    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b857b2-4411-480a-a755-699c535ccb25" containerName="payload-processing"
Oct 30 13:50:27.745960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745836    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6244f0-a709-46d6-bafd-dbad9edb2d74" containerName="maas-api"
Oct 30 13:50:27.745960 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.745850    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy"
Oct 30 13:50:27.751647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.751618    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.769376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.769342    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"]
Oct 30 13:50:27.777326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.777297    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-517a25\"/\"rate-test-model-517a25-kserve-self-signed-certs\""
Oct 30 13:50:27.777543 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.777436    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-517a25\"/\"default-dockercfg-spgj5\""
Oct 30 13:50:27.781671 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.781638    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-517a25\"/\"openshift-service-ca.crt\""
Oct 30 13:50:27.781848 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.781642    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-517a25\"/\"kube-root-ca.crt\""
Oct 30 13:50:27.811346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.811271    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-model-cache\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.811538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.811429    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-home\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.811538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.811451    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/171bd558-2e74-4d2c-ad34-41118c9c26ef-tls-certs\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.811538 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.811489    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8bhd\" (UniqueName: \"kubernetes.io/projected/171bd558-2e74-4d2c-ad34-41118c9c26ef-kube-api-access-z8bhd\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.811778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.811753    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-dshm\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.811840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.811787    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-tmp-dir\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.916003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.914422    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-dshm\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.916003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.914474    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-tmp-dir\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.916003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.914527    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-model-cache\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.916003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.914634    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-home\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.916003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.914668    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/171bd558-2e74-4d2c-ad34-41118c9c26ef-tls-certs\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.916003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.914721    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8bhd\" (UniqueName: \"kubernetes.io/projected/171bd558-2e74-4d2c-ad34-41118c9c26ef-kube-api-access-z8bhd\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.916003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.915594    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-model-cache\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.916003 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.915600    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-home\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.932158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.920645    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-dshm\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.932158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.920929    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-tmp-dir\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.934216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.934186    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/171bd558-2e74-4d2c-ad34-41118c9c26ef-tls-certs\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:27.941773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:27.941739    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8bhd\" (UniqueName: \"kubernetes.io/projected/171bd558-2e74-4d2c-ad34-41118c9c26ef-kube-api-access-z8bhd\") pod \"rate-test-model-517a25-kserve-5795466769-rssng\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") " pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:28.073414 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:28.073319    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:28.388754 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:50:28.388717    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod171bd558_2e74_4d2c_ad34_41118c9c26ef.slice/crio-b82dc2d6e43fc758c9bab5c317da94cd718afdf2a8a94c4daf1cf4ad63eac90f WatchSource:0}: Error finding container b82dc2d6e43fc758c9bab5c317da94cd718afdf2a8a94c4daf1cf4ad63eac90f: Status 404 returned error can't find the container with id b82dc2d6e43fc758c9bab5c317da94cd718afdf2a8a94c4daf1cf4ad63eac90f
Oct 30 13:50:28.389255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:28.389230    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"]
Oct 30 13:50:29.170713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:29.170669    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng" event={"ID":"171bd558-2e74-4d2c-ad34-41118c9c26ef","Type":"ContainerStarted","Data":"c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1"}
Oct 30 13:50:29.170713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:29.170720    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng" event={"ID":"171bd558-2e74-4d2c-ad34-41118c9c26ef","Type":"ContainerStarted","Data":"b82dc2d6e43fc758c9bab5c317da94cd718afdf2a8a94c4daf1cf4ad63eac90f"}
Oct 30 13:50:29.215501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:29.215450    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng" podStartSLOduration=2.215433453 podStartE2EDuration="2.215433453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:29.215102683 +0000 UTC m=+4687.270596523" watchObservedRunningTime="2026-10-30 13:50:29.215433453 +0000 UTC m=+4687.270927278"
Oct 30 13:50:30.076578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:30.076547    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:33.233549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:33.233510    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w" podUID="7f992cba-9c34-45a5-abc1-bfa307a8badb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:50:38.076978 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:38.076947    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:38.086955 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:38.086932    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:38.275136 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:38.275108    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:50:42.373874 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:50:42.373837    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:50:43.234342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:43.234279    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w" podUID="7f992cba-9c34-45a5-abc1-bfa307a8badb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:50:46.700983 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:46.700951    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:50:46.785215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:46.785164    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f992cba-9c34-45a5-abc1-bfa307a8badb-kube-api-access-r5n9t\" (UniqueName: \"kubernetes.io/projected/7f992cba-9c34-45a5-abc1-bfa307a8badb-kube-api-access-r5n9t\") pod \"7f992cba-9c34-45a5-abc1-bfa307a8badb\" (UID: \"7f992cba-9c34-45a5-abc1-bfa307a8badb\") "
Oct 30 13:50:46.785540 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:46.785459    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f992cba-9c34-45a5-abc1-bfa307a8badb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f992cba-9c34-45a5-abc1-bfa307a8badb-plugins-config-volume\") pod \"7f992cba-9c34-45a5-abc1-bfa307a8badb\" (UID: \"7f992cba-9c34-45a5-abc1-bfa307a8badb\") "
Oct 30 13:50:46.786058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:46.786004    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f992cba-9c34-45a5-abc1-bfa307a8badb-plugins-config-volume" pod "7f992cba-9c34-45a5-abc1-bfa307a8badb" (UID: "7f992cba-9c34-45a5-abc1-bfa307a8badb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:46.786418 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:46.786396    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f992cba-9c34-45a5-abc1-bfa307a8badb-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:46.788237 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:46.788204    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f992cba-9c34-45a5-abc1-bfa307a8badb-kube-api-access-r5n9t" pod "7f992cba-9c34-45a5-abc1-bfa307a8badb" (UID: "7f992cba-9c34-45a5-abc1-bfa307a8badb"). InnerVolumeSpecName "kube-api-access-r5n9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:46.888055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:46.888014    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5n9t\" (UniqueName: \"kubernetes.io/projected/7f992cba-9c34-45a5-abc1-bfa307a8badb-kube-api-access-r5n9t\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:47.329201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.329151    4429 generic.go:356] "Generic (PLEG): container finished" podID="7f992cba-9c34-45a5-abc1-bfa307a8badb" containerID="3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058" exitCode=0
Oct 30 13:50:47.329425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.329206    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"
Oct 30 13:50:47.329425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.329240    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w" event={"ID":"7f992cba-9c34-45a5-abc1-bfa307a8badb","Type":"ContainerDied","Data":"3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058"}
Oct 30 13:50:47.329425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.329317    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w" event={"ID":"7f992cba-9c34-45a5-abc1-bfa307a8badb","Type":"ContainerDied","Data":"e70b9f218f24f32c5ed13d25bb8bf0cd6ffd44f826d69388b7ae216ac79a1e58"}
Oct 30 13:50:47.329425 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.329351    4429 scope.go:122] "RemoveContainer" containerID="3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058"
Oct 30 13:50:47.342355 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.342326    4429 scope.go:122] "RemoveContainer" containerID="3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058"
Oct 30 13:50:47.342746 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:50:47.342723    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058\": container with ID starting with 3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058 not found: ID does not exist" containerID="3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058"
Oct 30 13:50:47.342802 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.342758    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058"} err="failed to get container status \"3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058\": rpc error: code = NotFound desc = could not find container \"3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058\": container with ID starting with 3c658b294578e70a4a62b7d1bfe9aad3948b7b8fa6b454fff66b6c3311ba3058 not found: ID does not exist"
Oct 30 13:50:47.430243 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.430204    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"]
Oct 30 13:50:47.443364 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:47.443316    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0d9115-f5cb4bd8b-6ts5w"]
Oct 30 13:50:48.077847 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:48.077800    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f992cba-9c34-45a5-abc1-bfa307a8badb" path="/var/lib/kubelet/pods/7f992cba-9c34-45a5-abc1-bfa307a8badb/volumes"
Oct 30 13:50:50.423082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.423041    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"]
Oct 30 13:50:50.423650 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.423367    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx" podUID="d5daadbb-275f-4346-820c-d9bbb818202c" containerName="maas-api" containerID="cri-o://59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181" gracePeriod=30
Oct 30 13:50:50.750385 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.750354    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:50:50.862670 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.862634    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5daadbb-275f-4346-820c-d9bbb818202c-kube-api-access-82xbj\" (UniqueName: \"kubernetes.io/projected/d5daadbb-275f-4346-820c-d9bbb818202c-kube-api-access-82xbj\") pod \"d5daadbb-275f-4346-820c-d9bbb818202c\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") "
Oct 30 13:50:50.863469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.863437    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-metrics-certs\") pod \"d5daadbb-275f-4346-820c-d9bbb818202c\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") "
Oct 30 13:50:50.863624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.863555    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5daadbb-275f-4346-820c-d9bbb818202c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5daadbb-275f-4346-820c-d9bbb818202c-openshift-service-ca\") pod \"d5daadbb-275f-4346-820c-d9bbb818202c\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") "
Oct 30 13:50:50.863698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.863680    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-maas-api-tls\") pod \"d5daadbb-275f-4346-820c-d9bbb818202c\" (UID: \"d5daadbb-275f-4346-820c-d9bbb818202c\") "
Oct 30 13:50:50.865201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.865170    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5daadbb-275f-4346-820c-d9bbb818202c-openshift-service-ca" pod "d5daadbb-275f-4346-820c-d9bbb818202c" (UID: "d5daadbb-275f-4346-820c-d9bbb818202c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:50.865900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.865871    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5daadbb-275f-4346-820c-d9bbb818202c-kube-api-access-82xbj" pod "d5daadbb-275f-4346-820c-d9bbb818202c" (UID: "d5daadbb-275f-4346-820c-d9bbb818202c"). InnerVolumeSpecName "kube-api-access-82xbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:50.867562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.867535    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-maas-api-tls" pod "d5daadbb-275f-4346-820c-d9bbb818202c" (UID: "d5daadbb-275f-4346-820c-d9bbb818202c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:50.885250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.885210    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-metrics-certs" pod "d5daadbb-275f-4346-820c-d9bbb818202c" (UID: "d5daadbb-275f-4346-820c-d9bbb818202c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:50.965979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.965948    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5daadbb-275f-4346-820c-d9bbb818202c-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:50.965979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.965980    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:50.965979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.965993    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82xbj\" (UniqueName: \"kubernetes.io/projected/d5daadbb-275f-4346-820c-d9bbb818202c-kube-api-access-82xbj\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:50.966255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:50.966005    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5daadbb-275f-4346-820c-d9bbb818202c-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:50:51.361587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.361546    4429 generic.go:356] "Generic (PLEG): container finished" podID="d5daadbb-275f-4346-820c-d9bbb818202c" containerID="59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181" exitCode=0
Oct 30 13:50:51.361833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.361606    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"
Oct 30 13:50:51.361833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.361634    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx" event={"ID":"d5daadbb-275f-4346-820c-d9bbb818202c","Type":"ContainerDied","Data":"59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181"}
Oct 30 13:50:51.361833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.361711    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx" event={"ID":"d5daadbb-275f-4346-820c-d9bbb818202c","Type":"ContainerDied","Data":"53c07ef92c69cf2b8bfd456ce6dbe1b10c4f7834b8ccffcbc7c469c451ad8994"}
Oct 30 13:50:51.361833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.361742    4429 scope.go:122] "RemoveContainer" containerID="59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181"
Oct 30 13:50:51.374457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.374432    4429 scope.go:122] "RemoveContainer" containerID="59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181"
Oct 30 13:50:51.375026 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:50:51.375005    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181\": container with ID starting with 59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181 not found: ID does not exist" containerID="59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181"
Oct 30 13:50:51.375112 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.375041    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181"} err="failed to get container status \"59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181\": rpc error: code = NotFound desc = could not find container \"59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181\": container with ID starting with 59819e9d4520675e86ae34d6b1f05fe238f96245a51e3ececf55d49a1c86d181 not found: ID does not exist"
Oct 30 13:50:51.420570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.420529    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"]
Oct 30 13:50:51.434661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:51.434619    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d01cd5a5-5b4bdfd444-fg5zx"]
Oct 30 13:50:52.076804 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:52.076768    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5daadbb-275f-4346-820c-d9bbb818202c" path="/var/lib/kubelet/pods/d5daadbb-275f-4346-820c-d9bbb818202c/volumes"
Oct 30 13:50:57.910906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.910873    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"]
Oct 30 13:50:57.911654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.911629    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d5daadbb-275f-4346-820c-d9bbb818202c" containerName="maas-api"
Oct 30 13:50:57.911654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.911649    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5daadbb-275f-4346-820c-d9bbb818202c" containerName="maas-api"
Oct 30 13:50:57.911654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.911661    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy"
Oct 30 13:50:57.911854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.911666    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f924a4-fb89-4052-bf95-790c1129da55" containerName="istio-proxy"
Oct 30 13:50:57.911854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.911685    4429 cpu_manager.go:414] "containerMap: removing container" podUID="7f992cba-9c34-45a5-abc1-bfa307a8badb" containerName="payload-processing"
Oct 30 13:50:57.911854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.911690    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f992cba-9c34-45a5-abc1-bfa307a8badb" containerName="payload-processing"
Oct 30 13:50:57.911854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.911802    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5daadbb-275f-4346-820c-d9bbb818202c" containerName="maas-api"
Oct 30 13:50:57.911854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.911814    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f992cba-9c34-45a5-abc1-bfa307a8badb" containerName="payload-processing"
Oct 30 13:50:57.917174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.917153    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:57.936759 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.936732    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"]
Oct 30 13:50:57.946744 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.946714    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-241ffa\"/\"openshift-service-ca.crt\""
Oct 30 13:50:57.946879 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.946802    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-241ffa\"/\"kube-root-ca.crt\""
Oct 30 13:50:57.949131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.949109    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-241ffa\"/\"default-dockercfg-rjfgf\""
Oct 30 13:50:57.949131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.949109    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-241ffa\"/\"rate-test-model-241ffa-kserve-self-signed-certs\""
Oct 30 13:50:57.996146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.996122    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-dshm\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:57.996278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.996194    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba1899f-c0d5-4484-aa5b-eb878b13b571-tls-certs\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:57.996416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.996391    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-tmp-dir\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:57.996643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.996623    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-model-cache\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:57.996726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.996668    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-home\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:57.996726 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:57.996706    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hc2c\" (UniqueName: \"kubernetes.io/projected/eba1899f-c0d5-4484-aa5b-eb878b13b571-kube-api-access-7hc2c\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.098427 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.098399    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-model-cache\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.098596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.098449    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-home\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.098596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.098486    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hc2c\" (UniqueName: \"kubernetes.io/projected/eba1899f-c0d5-4484-aa5b-eb878b13b571-kube-api-access-7hc2c\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.098596 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.098523    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-dshm\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.098764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.098601    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba1899f-c0d5-4484-aa5b-eb878b13b571-tls-certs\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.098764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.098678    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-tmp-dir\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.098871 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.098844    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-model-cache\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.098927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.098851    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-home\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.099133 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.099114    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-tmp-dir\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.101196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.101174    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-dshm\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.101474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.101457    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba1899f-c0d5-4484-aa5b-eb878b13b571-tls-certs\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.124692 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.124668    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hc2c\" (UniqueName: \"kubernetes.io/projected/eba1899f-c0d5-4484-aa5b-eb878b13b571-kube-api-access-7hc2c\") pod \"rate-test-model-241ffa-kserve-6549564bc7-lhgw6\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") " pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.237365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.237334    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:50:58.546416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:58.546385    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"]
Oct 30 13:50:59.428640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:59.428602    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6" event={"ID":"eba1899f-c0d5-4484-aa5b-eb878b13b571","Type":"ContainerStarted","Data":"3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e"}
Oct 30 13:50:59.429145 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:59.428660    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6" event={"ID":"eba1899f-c0d5-4484-aa5b-eb878b13b571","Type":"ContainerStarted","Data":"c14e9b3bae3a6279b0287ae3ed055657f775ed8819864374af93b14170a4d146"}
Oct 30 13:50:59.497390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:50:59.497341    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6" podStartSLOduration=2.497325328 podStartE2EDuration="2.497325328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:59.494837173 +0000 UTC m=+4717.550330998" watchObservedRunningTime="2026-10-30 13:50:59.497325328 +0000 UTC m=+4717.552819156"
Oct 30 13:51:00.238062 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:00.238029    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:51:08.238608 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:08.238524    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:51:08.251333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:08.251277    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:51:08.508320 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:08.508218    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:51:12.374323 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:51:12.374258    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:51:24.401980 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:24.401946    4429 scope.go:122] "RemoveContainer" containerID="d565912c0d6814fd1955eac172b9a28bf8eb94f239811caf22041038d788c5ed"
Oct 30 13:51:40.553197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.553153    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"]
Oct 30 13:51:40.558175 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.558152    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.578101 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.578071    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"]
Oct 30 13:51:40.587841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.587811    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-451a21\"/\"default-dockercfg-g5zjv\""
Oct 30 13:51:40.588058 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.588040    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-451a21\"/\"kube-root-ca.crt\""
Oct 30 13:51:40.589018 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.588999    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-451a21\"/\"openshift-service-ca.crt\""
Oct 30 13:51:40.589317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.589295    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-451a21\"/\"rate-test-model-451a21-kserve-self-signed-certs\""
Oct 30 13:51:40.601113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.601091    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-tmp-dir\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.601273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.601252    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9830a470-d928-4baf-b5b0-da84535d7cd8-tls-certs\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.601392 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.601375    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-home\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.601575 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.601556    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-model-cache\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.601651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.601589    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-dshm\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.601731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.601656    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw85v\" (UniqueName: \"kubernetes.io/projected/9830a470-d928-4baf-b5b0-da84535d7cd8-kube-api-access-qw85v\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.703478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.703445    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9830a470-d928-4baf-b5b0-da84535d7cd8-tls-certs\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.703683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.703495    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-home\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.703683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.703595    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-model-cache\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.703683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.703630    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-dshm\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.703860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.703689    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw85v\" (UniqueName: \"kubernetes.io/projected/9830a470-d928-4baf-b5b0-da84535d7cd8-kube-api-access-qw85v\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.703860 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.703770    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-tmp-dir\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.704055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.704030    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-home\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.704055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.704071    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-model-cache\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.704409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.704260    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-tmp-dir\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.706646 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.706623    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-dshm\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.706872 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.706855    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9830a470-d928-4baf-b5b0-da84535d7cd8-tls-certs\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.728903 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.728876    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw85v\" (UniqueName: \"kubernetes.io/projected/9830a470-d928-4baf-b5b0-da84535d7cd8-kube-api-access-qw85v\") pod \"rate-test-model-451a21-kserve-78b74dc557-2h79m\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") " pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:40.873336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:40.873200    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:41.179688 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:51:41.179647    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9830a470_d928_4baf_b5b0_da84535d7cd8.slice/crio-24cb51c9c57aad4ff387b230b5108048a7528a36b250a4776f931772a49501e4 WatchSource:0}: Error finding container 24cb51c9c57aad4ff387b230b5108048a7528a36b250a4776f931772a49501e4: Status 404 returned error can't find the container with id 24cb51c9c57aad4ff387b230b5108048a7528a36b250a4776f931772a49501e4
Oct 30 13:51:41.181039 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:41.181013    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"]
Oct 30 13:51:41.734512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:41.734469    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m" event={"ID":"9830a470-d928-4baf-b5b0-da84535d7cd8","Type":"ContainerStarted","Data":"cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e"}
Oct 30 13:51:41.735042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:41.734523    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m" event={"ID":"9830a470-d928-4baf-b5b0-da84535d7cd8","Type":"ContainerStarted","Data":"24cb51c9c57aad4ff387b230b5108048a7528a36b250a4776f931772a49501e4"}
Oct 30 13:51:41.781696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:41.781626    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m" podStartSLOduration=1.781610433 podStartE2EDuration="1.781610433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:41.779690312 +0000 UTC m=+4759.835184160" watchObservedRunningTime="2026-10-30 13:51:41.781610433 +0000 UTC m=+4759.837104292"
Oct 30 13:51:41.873620 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:41.873591    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:42.373063 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:51:42.373028    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:51:50.873829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:50.873794    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:50.887261 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:50.887229    4429 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:51.818455 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:51.818428    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:51:57.476152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:57.476114    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"]
Oct 30 13:51:57.476682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:51:57.476360    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" containerID="cri-o://491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9" gracePeriod=30
Oct 30 13:52:00.445795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:00.445742    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:52:00.446365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:00.445815    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:52:02.772299 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.772250    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:52:02.853852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.853822    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-credential-socket\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.854045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.854009    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-podinfo\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.854113 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.854083    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-envoy\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.854177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.854135    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-certs\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.854249 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.854230    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-kube-api-access-k9vtv\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-kube-api-access-k9vtv\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.854346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.854308    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-data\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.854417 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.854348    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istiod-ca-cert\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.854417 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.854387    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-token\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-token\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.854524 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.854451    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-socket\") pod \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\" (UID: \"153040de-ee6d-4eb0-9333-5e7d0c36dc60\") "
Oct 30 13:52:02.855344 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.855314    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-certs" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:02.855766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.855736    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-data" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:02.855968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.855943    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-socket" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:02.856152 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.856127    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istiod-ca-cert" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:02.856223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.856194    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-credential-socket" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:02.859067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.859032    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-podinfo" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:52:02.859996 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.859966    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-token" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:02.860098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.860053    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-kube-api-access-k9vtv" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "kube-api-access-k9vtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:02.860403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.860379    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-envoy" pod "153040de-ee6d-4eb0-9333-5e7d0c36dc60" (UID: "153040de-ee6d-4eb0-9333-5e7d0c36dc60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:52:02.893402 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.893370    4429 generic.go:356] "Generic (PLEG): container finished" podID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerID="491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9" exitCode=0
Oct 30 13:52:02.893570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.893434    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"
Oct 30 13:52:02.893570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.893449    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" event={"ID":"153040de-ee6d-4eb0-9333-5e7d0c36dc60","Type":"ContainerDied","Data":"491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9"}
Oct 30 13:52:02.893570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.893503    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z" event={"ID":"153040de-ee6d-4eb0-9333-5e7d0c36dc60","Type":"ContainerDied","Data":"899a2c646f8e7c1edc83f21d0181a16cd146a193ec98604b5c534b42e5ebd28d"}
Oct 30 13:52:02.893570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.893527    4429 scope.go:122] "RemoveContainer" containerID="491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9"
Oct 30 13:52:02.904711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.904695    4429 scope.go:122] "RemoveContainer" containerID="7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0"
Oct 30 13:52:02.914489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.914466    4429 scope.go:122] "RemoveContainer" containerID="491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9"
Oct 30 13:52:02.914817 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:52:02.914783    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9\": container with ID starting with 491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9 not found: ID does not exist" containerID="491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9"
Oct 30 13:52:02.914897 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.914823    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9"} err="failed to get container status \"491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9\": rpc error: code = NotFound desc = could not find container \"491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9\": container with ID starting with 491f6670fa72ecfa65f2c93279d1996f8c122e836cb6e983951b49c7fad959e9 not found: ID does not exist"
Oct 30 13:52:02.914897 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.914845    4429 scope.go:122] "RemoveContainer" containerID="7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0"
Oct 30 13:52:02.915102 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:52:02.915077    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0\": container with ID starting with 7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0 not found: ID does not exist" containerID="7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0"
Oct 30 13:52:02.915208 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.915102    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0"} err="failed to get container status \"7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0\": rpc error: code = NotFound desc = could not find container \"7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0\": container with ID starting with 7a5f0afa104e618a449db84ece99d783f4560b32690afb64e66fd5abbbe31ed0 not found: ID does not exist"
Oct 30 13:52:02.952124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.952098    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"]
Oct 30 13:52:02.956539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956509    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.956539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956539    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.956777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956580    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.956777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956598    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vtv\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-kube-api-access-k9vtv\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.956777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956613    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.956777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956638    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.956777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956647    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153040de-ee6d-4eb0-9333-5e7d0c36dc60-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.956777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956660    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.956777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.956668    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153040de-ee6d-4eb0-9333-5e7d0c36dc60-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:02.958473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:02.958454    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-bd2aaf-openshift-default-849cb87f69-w524z"]
Oct 30 13:52:04.075916 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:04.075884    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" path="/var/lib/kubelet/pods/153040de-ee6d-4eb0-9333-5e7d0c36dc60/volumes"
Oct 30 13:52:08.642559 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:08.642523    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"]
Oct 30 13:52:12.374372 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:52:12.374332    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:52:13.643087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:13.643040    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw" podUID="62d4287c-8aba-4241-9d46-66577c5dc2b6" containerName="payload-pre-processing" containerID="cri-o://b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb" gracePeriod=25
Oct 30 13:52:23.324460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:23.324415    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw" podUID="62d4287c-8aba-4241-9d46-66577c5dc2b6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:52:33.324863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:33.324773    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw" podUID="62d4287c-8aba-4241-9d46-66577c5dc2b6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:52:33.933149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:33.933126    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:52:34.004471 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.004440    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d4287c-8aba-4241-9d46-66577c5dc2b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d4287c-8aba-4241-9d46-66577c5dc2b6-plugins-config-volume\") pod \"62d4287c-8aba-4241-9d46-66577c5dc2b6\" (UID: \"62d4287c-8aba-4241-9d46-66577c5dc2b6\") "
Oct 30 13:52:34.004694 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.004679    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d4287c-8aba-4241-9d46-66577c5dc2b6-kube-api-access-nqjhl\" (UniqueName: \"kubernetes.io/projected/62d4287c-8aba-4241-9d46-66577c5dc2b6-kube-api-access-nqjhl\") pod \"62d4287c-8aba-4241-9d46-66577c5dc2b6\" (UID: \"62d4287c-8aba-4241-9d46-66577c5dc2b6\") "
Oct 30 13:52:34.004786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.004765    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d4287c-8aba-4241-9d46-66577c5dc2b6-plugins-config-volume" pod "62d4287c-8aba-4241-9d46-66577c5dc2b6" (UID: "62d4287c-8aba-4241-9d46-66577c5dc2b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:34.005115 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.005098    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d4287c-8aba-4241-9d46-66577c5dc2b6-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:34.006903 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.006874    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d4287c-8aba-4241-9d46-66577c5dc2b6-kube-api-access-nqjhl" pod "62d4287c-8aba-4241-9d46-66577c5dc2b6" (UID: "62d4287c-8aba-4241-9d46-66577c5dc2b6"). InnerVolumeSpecName "kube-api-access-nqjhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:34.106679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.106641    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqjhl\" (UniqueName: \"kubernetes.io/projected/62d4287c-8aba-4241-9d46-66577c5dc2b6-kube-api-access-nqjhl\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:34.136406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.136368    4429 generic.go:356] "Generic (PLEG): container finished" podID="62d4287c-8aba-4241-9d46-66577c5dc2b6" containerID="b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb" exitCode=0
Oct 30 13:52:34.136564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.136425    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"
Oct 30 13:52:34.136564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.136451    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw" event={"ID":"62d4287c-8aba-4241-9d46-66577c5dc2b6","Type":"ContainerDied","Data":"b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb"}
Oct 30 13:52:34.136564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.136507    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw" event={"ID":"62d4287c-8aba-4241-9d46-66577c5dc2b6","Type":"ContainerDied","Data":"55c27076d3e69f1251eecc2613ea9967abd697fae5f334e9824b44b5f71c1b87"}
Oct 30 13:52:34.136564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.136528    4429 scope.go:122] "RemoveContainer" containerID="b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb"
Oct 30 13:52:34.146765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.146745    4429 scope.go:122] "RemoveContainer" containerID="b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb"
Oct 30 13:52:34.147042 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:52:34.147022    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb\": container with ID starting with b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb not found: ID does not exist" containerID="b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb"
Oct 30 13:52:34.147103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.147061    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb"} err="failed to get container status \"b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb\": rpc error: code = NotFound desc = could not find container \"b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb\": container with ID starting with b027bffcde3011ad04b09958ba463914347666454aad75aa438e60da6702e2eb not found: ID does not exist"
Oct 30 13:52:34.196801 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.196774    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"]
Oct 30 13:52:34.200863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:34.200843    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71fbeb-d4dfbc4ff-g5ppw"]
Oct 30 13:52:36.075908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:36.075863    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d4287c-8aba-4241-9d46-66577c5dc2b6" path="/var/lib/kubelet/pods/62d4287c-8aba-4241-9d46-66577c5dc2b6/volumes"
Oct 30 13:52:39.332317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.332259    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"]
Oct 30 13:52:39.332784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.332553    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb" podUID="60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" containerName="maas-api" containerID="cri-o://71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc" gracePeriod=30
Oct 30 13:52:39.639149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.639113    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:52:39.681084 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.681048    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-metrics-certs\") pod \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") "
Oct 30 13:52:39.681333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.681256    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls\") pod \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") "
Oct 30 13:52:39.681433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.681388    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-openshift-service-ca\") pod \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") "
Oct 30 13:52:39.681433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.681422    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-kube-api-access-k5l9w\" (UniqueName: \"kubernetes.io/projected/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-kube-api-access-k5l9w\") pod \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\" (UID: \"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9\") "
Oct 30 13:52:39.681809 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.681779    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-openshift-service-ca" pod "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" (UID: "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:52:39.682219 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.682193    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:39.683628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.683603    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls" pod "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" (UID: "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:39.683869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.683846    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-kube-api-access-k5l9w" pod "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" (UID: "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9"). InnerVolumeSpecName "kube-api-access-k5l9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:52:39.686841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.686807    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-metrics-certs" pod "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" (UID: "60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:52:39.783763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.783729    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5l9w\" (UniqueName: \"kubernetes.io/projected/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-kube-api-access-k5l9w\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:39.783763 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.783761    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:39.783977 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:39.783775    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:52:40.181412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.181375    4429 generic.go:356] "Generic (PLEG): container finished" podID="60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" containerID="71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc" exitCode=0
Oct 30 13:52:40.181625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.181441    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"
Oct 30 13:52:40.181625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.181466    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb" event={"ID":"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9","Type":"ContainerDied","Data":"71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc"}
Oct 30 13:52:40.181625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.181533    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb" event={"ID":"60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9","Type":"ContainerDied","Data":"1bc84a4e76676bf90d2ad716eebd76f31a1ee7f75cdfc23a44234b142534ce3d"}
Oct 30 13:52:40.181625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.181564    4429 scope.go:122] "RemoveContainer" containerID="71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc"
Oct 30 13:52:40.197912 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.197889    4429 scope.go:122] "RemoveContainer" containerID="71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc"
Oct 30 13:52:40.198524 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:52:40.198489    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc\": container with ID starting with 71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc not found: ID does not exist" containerID="71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc"
Oct 30 13:52:40.198684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.198529    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc"} err="failed to get container status \"71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc\": rpc error: code = NotFound desc = could not find container \"71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc\": container with ID starting with 71807dd9bcc741bb8b4b1eda06ab105c55a71a5ba77a83a7c2aa3ca0c570b2cc not found: ID does not exist"
Oct 30 13:52:40.241534 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.241501    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"]
Oct 30 13:52:40.269359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:40.269331    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb"]
Oct 30 13:52:42.076811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:42.076775    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" path="/var/lib/kubelet/pods/60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9/volumes"
Oct 30 13:52:42.373257 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:52:42.373175    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:52:42.544958 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:42.544913    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71fbeb-644f87df4b-kftpb" podUID="60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.158:8443/health\": context deadline exceeded"
Oct 30 13:52:53.377564 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:53.377520    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"]
Oct 30 13:52:53.400109 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:53.400072    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"]
Oct 30 13:52:53.418853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:52:53.418810    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"]
Oct 30 13:53:08.400275 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.400227    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" containerID="cri-o://907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e" gracePeriod=45
Oct 30 13:53:08.422468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.422436    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" containerID="cri-o://f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a" gracePeriod=45
Oct 30 13:53:08.441935 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.441879    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main" containerID="cri-o://faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790" gracePeriod=45
Oct 30 13:53:08.740778 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.739870    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:53:08.807268 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.807244    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:53:08.810915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.810876    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:53:08.898841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898424    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tls-certs\") pod \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") "
Oct 30 13:53:08.898841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898495    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-kube-api-access-s2knk\" (UniqueName: \"kubernetes.io/projected/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-kube-api-access-s2knk\") pod \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") "
Oct 30 13:53:08.898841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898527    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1494f395-8b0b-472c-bf57-645835078d44-tls-certs\" (UniqueName: \"kubernetes.io/secret/1494f395-8b0b-472c-bf57-645835078d44-tls-certs\") pod \"1494f395-8b0b-472c-bf57-645835078d44\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") "
Oct 30 13:53:08.898841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898592    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-model-cache\") pod \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") "
Oct 30 13:53:08.898841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898623    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-dshm\") pod \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") "
Oct 30 13:53:08.898841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898694    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-home\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-home\") pod \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") "
Oct 30 13:53:08.898841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898783    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-home\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-home\") pod \"1494f395-8b0b-472c-bf57-645835078d44\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") "
Oct 30 13:53:08.898841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898817    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tls-certs\") pod \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898950    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tmp-dir\") pod \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.898993    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-dshm\") pod \"1494f395-8b0b-472c-bf57-645835078d44\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.899061    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1494f395-8b0b-472c-bf57-645835078d44-kube-api-access-phlvn\" (UniqueName: \"kubernetes.io/projected/1494f395-8b0b-472c-bf57-645835078d44-kube-api-access-phlvn\") pod \"1494f395-8b0b-472c-bf57-645835078d44\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.899121    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tmp-dir\") pod \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.899187    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-dshm\") pod \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.899211    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-tmp-dir\") pod \"1494f395-8b0b-472c-bf57-645835078d44\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.899254    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-home\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-home\") pod \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\" (UID: \"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.899299    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0507ec-f598-4e58-ad65-2a0afb295eaf-kube-api-access-z2ftx\" (UniqueName: \"kubernetes.io/projected/2c0507ec-f598-4e58-ad65-2a0afb295eaf-kube-api-access-z2ftx\") pod \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.899337    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-model-cache\") pod \"1494f395-8b0b-472c-bf57-645835078d44\" (UID: \"1494f395-8b0b-472c-bf57-645835078d44\") "
Oct 30 13:53:08.899436 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.899410    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-model-cache\") pod \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\" (UID: \"2c0507ec-f598-4e58-ad65-2a0afb295eaf\") "
Oct 30 13:53:08.902825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.900107    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-home" pod "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" (UID: "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.902825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.900279    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-model-cache" pod "2c0507ec-f598-4e58-ad65-2a0afb295eaf" (UID: "2c0507ec-f598-4e58-ad65-2a0afb295eaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.902825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.900337    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-tmp-dir" pod "1494f395-8b0b-472c-bf57-645835078d44" (UID: "1494f395-8b0b-472c-bf57-645835078d44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.902825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.900517    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-model-cache" pod "1494f395-8b0b-472c-bf57-645835078d44" (UID: "1494f395-8b0b-472c-bf57-645835078d44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.902825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.900753    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tmp-dir" pod "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" (UID: "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.902825 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.902517    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-home" pod "1494f395-8b0b-472c-bf57-645835078d44" (UID: "1494f395-8b0b-472c-bf57-645835078d44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.903691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.903485    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-home" pod "2c0507ec-f598-4e58-ad65-2a0afb295eaf" (UID: "2c0507ec-f598-4e58-ad65-2a0afb295eaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.903691 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.903635    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0507ec-f598-4e58-ad65-2a0afb295eaf-kube-api-access-z2ftx" pod "2c0507ec-f598-4e58-ad65-2a0afb295eaf" (UID: "2c0507ec-f598-4e58-ad65-2a0afb295eaf"). InnerVolumeSpecName "kube-api-access-z2ftx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:08.904214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.904192    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-model-cache" pod "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" (UID: "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.904460 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.904404    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tmp-dir" pod "2c0507ec-f598-4e58-ad65-2a0afb295eaf" (UID: "2c0507ec-f598-4e58-ad65-2a0afb295eaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.910685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.910608    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tls-certs" pod "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" (UID: "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:08.910685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.910556    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-dshm" pod "1494f395-8b0b-472c-bf57-645835078d44" (UID: "1494f395-8b0b-472c-bf57-645835078d44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.911264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.911226    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1494f395-8b0b-472c-bf57-645835078d44-tls-certs" pod "1494f395-8b0b-472c-bf57-645835078d44" (UID: "1494f395-8b0b-472c-bf57-645835078d44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:08.911393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.911261    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1494f395-8b0b-472c-bf57-645835078d44-kube-api-access-phlvn" pod "1494f395-8b0b-472c-bf57-645835078d44" (UID: "1494f395-8b0b-472c-bf57-645835078d44"). InnerVolumeSpecName "kube-api-access-phlvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:08.911469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.911420    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tls-certs" pod "2c0507ec-f598-4e58-ad65-2a0afb295eaf" (UID: "2c0507ec-f598-4e58-ad65-2a0afb295eaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:08.911532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.911470    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-kube-api-access-s2knk" pod "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" (UID: "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1"). InnerVolumeSpecName "kube-api-access-s2knk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:08.911690 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.911667    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-dshm" pod "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" (UID: "7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:08.911927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:08.911908    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-dshm" pod "2c0507ec-f598-4e58-ad65-2a0afb295eaf" (UID: "2c0507ec-f598-4e58-ad65-2a0afb295eaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:09.001523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001489    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2knk\" (UniqueName: \"kubernetes.io/projected/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-kube-api-access-s2knk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001524    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1494f395-8b0b-472c-bf57-645835078d44-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001541    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001555    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001568    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001580    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001593    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001606    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001619    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001632    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phlvn\" (UniqueName: \"kubernetes.io/projected/1494f395-8b0b-472c-bf57-645835078d44-kube-api-access-phlvn\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001646    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001755    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001768    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001781    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001795    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2ftx\" (UniqueName: \"kubernetes.io/projected/2c0507ec-f598-4e58-ad65-2a0afb295eaf-kube-api-access-z2ftx\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001810    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1494f395-8b0b-472c-bf57-645835078d44-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001824    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c0507ec-f598-4e58-ad65-2a0afb295eaf-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.001873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.001837    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:09.415625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.415589    4429 generic.go:356] "Generic (PLEG): container finished" podID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerID="907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e" exitCode=0
Oct 30 13:53:09.416114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.415651    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"
Oct 30 13:53:09.416114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.415682    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" event={"ID":"2c0507ec-f598-4e58-ad65-2a0afb295eaf","Type":"ContainerDied","Data":"907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e"}
Oct 30 13:53:09.416114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.415734    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" event={"ID":"2c0507ec-f598-4e58-ad65-2a0afb295eaf","Type":"ContainerDied","Data":"7bcbe55cec4c3d80d27c1431162b6eb59a9fba02e6ebda53958641b8ab90166e"}
Oct 30 13:53:09.416114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.415759    4429 scope.go:122] "RemoveContainer" containerID="907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e"
Oct 30 13:53:09.418106 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.418077    4429 generic.go:356] "Generic (PLEG): container finished" podID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerID="f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a" exitCode=0
Oct 30 13:53:09.418378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.418306    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" event={"ID":"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1","Type":"ContainerDied","Data":"f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a"}
Oct 30 13:53:09.418378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.418346    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" event={"ID":"7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1","Type":"ContainerDied","Data":"244a9fc334a00fb7f71d4081953d4e83132b2fc3582ac2a32f9d8c45e2ca99d7"}
Oct 30 13:53:09.418631 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.418612    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"
Oct 30 13:53:09.420081 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.420052    4429 generic.go:356] "Generic (PLEG): container finished" podID="1494f395-8b0b-472c-bf57-645835078d44" containerID="faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790" exitCode=0
Oct 30 13:53:09.420201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.420099    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" event={"ID":"1494f395-8b0b-472c-bf57-645835078d44","Type":"ContainerDied","Data":"faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790"}
Oct 30 13:53:09.420201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.420124    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs" event={"ID":"1494f395-8b0b-472c-bf57-645835078d44","Type":"ContainerDied","Data":"8b9c6093f1cd0ccd82765ad1a714269d5b20c671b5b22f3716aebd91dac71a43"}
Oct 30 13:53:09.420201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.420181    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"
Oct 30 13:53:09.437432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.437407    4429 scope.go:122] "RemoveContainer" containerID="907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e"
Oct 30 13:53:09.437756 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:53:09.437739    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e\": container with ID starting with 907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e not found: ID does not exist" containerID="907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e"
Oct 30 13:53:09.437820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.437764    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e"} err="failed to get container status \"907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e\": rpc error: code = NotFound desc = could not find container \"907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e\": container with ID starting with 907ab49c34ec1add0b24706260fdac53dc01829e1ee6c909cb58e01efb88476e not found: ID does not exist"
Oct 30 13:53:09.437820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.437781    4429 scope.go:122] "RemoveContainer" containerID="f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a"
Oct 30 13:53:09.447766 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.447740    4429 scope.go:122] "RemoveContainer" containerID="f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a"
Oct 30 13:53:09.448102 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:53:09.448074    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a\": container with ID starting with f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a not found: ID does not exist" containerID="f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a"
Oct 30 13:53:09.448200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.448109    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a"} err="failed to get container status \"f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a\": rpc error: code = NotFound desc = could not find container \"f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a\": container with ID starting with f0a8cb7bd369f7bf34b03d03ad6ec822115dd624325ed28fcf76a06af960fd7a not found: ID does not exist"
Oct 30 13:53:09.448200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.448125    4429 scope.go:122] "RemoveContainer" containerID="faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790"
Oct 30 13:53:09.457521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.457497    4429 scope.go:122] "RemoveContainer" containerID="faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790"
Oct 30 13:53:09.457823 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:53:09.457805    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790\": container with ID starting with faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790 not found: ID does not exist" containerID="faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790"
Oct 30 13:53:09.457899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.457833    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790"} err="failed to get container status \"faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790\": rpc error: code = NotFound desc = could not find container \"faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790\": container with ID starting with faf584bd9d93d8e03b57c61654ee07222556abccafaacfe54a8e50cc0ef26790 not found: ID does not exist"
Oct 30 13:53:09.506967 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.506938    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"]
Oct 30 13:53:09.521852 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.521819    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd"]
Oct 30 13:53:09.547547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.547515    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"]
Oct 30 13:53:09.561083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.561059    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/simulator-w1-6ff0e6-kserve-86878c6495-mklxs"]
Oct 30 13:53:09.594367 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.594332    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"]
Oct 30 13:53:09.608099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.608064    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj"]
Oct 30 13:53:09.674979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.674893    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/distinct-w1-6ff0e6-kserve-6679d8d8cc-rp6mj" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:53:09.735604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:09.735562    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ff0e6/premium-w1-6ff0e6-kserve-646d49b58-ncntd" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:53:10.077268 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:10.077234    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1494f395-8b0b-472c-bf57-645835078d44" path="/var/lib/kubelet/pods/1494f395-8b0b-472c-bf57-645835078d44/volumes"
Oct 30 13:53:10.077769 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:10.077752    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" path="/var/lib/kubelet/pods/2c0507ec-f598-4e58-ad65-2a0afb295eaf/volumes"
Oct 30 13:53:10.078172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:10.078159    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" path="/var/lib/kubelet/pods/7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1/volumes"
Oct 30 13:53:12.372596 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:53:12.372553    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:53:36.306908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.306872    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"]
Oct 30 13:53:36.307449 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.307142    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m" podUID="88228612-d786-4f85-9cea-076c8db124a4" containerName="maas-api" containerID="cri-o://583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae" gracePeriod=30
Oct 30 13:53:36.591629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.591604    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:53:36.622001 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.621968    4429 generic.go:356] "Generic (PLEG): container finished" podID="88228612-d786-4f85-9cea-076c8db124a4" containerID="583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae" exitCode=0
Oct 30 13:53:36.622190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.622013    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"
Oct 30 13:53:36.622190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.622053    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m" event={"ID":"88228612-d786-4f85-9cea-076c8db124a4","Type":"ContainerDied","Data":"583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae"}
Oct 30 13:53:36.622190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.622117    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m" event={"ID":"88228612-d786-4f85-9cea-076c8db124a4","Type":"ContainerDied","Data":"f241c44d0055ac2e097242e628f62d86ae99d971b20f28e9ee17286608b11a1f"}
Oct 30 13:53:36.622190 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.622146    4429 scope.go:122] "RemoveContainer" containerID="583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae"
Oct 30 13:53:36.632327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.632312    4429 scope.go:122] "RemoveContainer" containerID="583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae"
Oct 30 13:53:36.632647 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:53:36.632629    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae\": container with ID starting with 583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae not found: ID does not exist" containerID="583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae"
Oct 30 13:53:36.632702 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.632656    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae"} err="failed to get container status \"583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae\": rpc error: code = NotFound desc = could not find container \"583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae\": container with ID starting with 583ed3f505749784d2ff2da9b467f1508b4f4a5a8c6a302611d0ba0e8c6449ae not found: ID does not exist"
Oct 30 13:53:36.757404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.757374    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-maas-api-tls\") pod \"88228612-d786-4f85-9cea-076c8db124a4\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") "
Oct 30 13:53:36.757624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.757464    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-metrics-certs\") pod \"88228612-d786-4f85-9cea-076c8db124a4\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") "
Oct 30 13:53:36.757624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.757509    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88228612-d786-4f85-9cea-076c8db124a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88228612-d786-4f85-9cea-076c8db124a4-openshift-service-ca\") pod \"88228612-d786-4f85-9cea-076c8db124a4\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") "
Oct 30 13:53:36.757760 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.757734    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88228612-d786-4f85-9cea-076c8db124a4-kube-api-access-pq9qc\" (UniqueName: \"kubernetes.io/projected/88228612-d786-4f85-9cea-076c8db124a4-kube-api-access-pq9qc\") pod \"88228612-d786-4f85-9cea-076c8db124a4\" (UID: \"88228612-d786-4f85-9cea-076c8db124a4\") "
Oct 30 13:53:36.759087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.758981    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88228612-d786-4f85-9cea-076c8db124a4-openshift-service-ca" pod "88228612-d786-4f85-9cea-076c8db124a4" (UID: "88228612-d786-4f85-9cea-076c8db124a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:36.760536 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.760496    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-maas-api-tls" pod "88228612-d786-4f85-9cea-076c8db124a4" (UID: "88228612-d786-4f85-9cea-076c8db124a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:36.760657 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.760584    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-metrics-certs" pod "88228612-d786-4f85-9cea-076c8db124a4" (UID: "88228612-d786-4f85-9cea-076c8db124a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:36.763014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.762992    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88228612-d786-4f85-9cea-076c8db124a4-kube-api-access-pq9qc" pod "88228612-d786-4f85-9cea-076c8db124a4" (UID: "88228612-d786-4f85-9cea-076c8db124a4"). InnerVolumeSpecName "kube-api-access-pq9qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:36.860267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.860174    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:36.860267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.860218    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88228612-d786-4f85-9cea-076c8db124a4-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:36.860267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.860233    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88228612-d786-4f85-9cea-076c8db124a4-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:36.860267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.860247    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq9qc\" (UniqueName: \"kubernetes.io/projected/88228612-d786-4f85-9cea-076c8db124a4-kube-api-access-pq9qc\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:53:36.981252 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.979957    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"]
Oct 30 13:53:36.996399 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:36.996361    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m"]
Oct 30 13:53:38.077820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:38.077787    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88228612-d786-4f85-9cea-076c8db124a4" path="/var/lib/kubelet/pods/88228612-d786-4f85-9cea-076c8db124a4/volumes"
Oct 30 13:53:39.542861 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:39.542819    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-acf9e2-5bbb68879c-whz6m" podUID="88228612-d786-4f85-9cea-076c8db124a4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.188:8443/health\": context deadline exceeded"
Oct 30 13:53:42.378116 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:53:42.377999    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:53:56.482711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:53:56.482679    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"]
Oct 30 13:54:01.483188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:01.483146    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w" podUID="d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" containerName="payload-processing" containerID="cri-o://5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf" gracePeriod=25
Oct 30 13:54:07.976207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:07.976120    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w" podUID="d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:54:08.530468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:08.530422    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"]
Oct 30 13:54:08.530723 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:08.530690    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy" containerID="cri-o://f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd" gracePeriod=30
Oct 30 13:54:12.373533 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:54:12.373494    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:54:13.814010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.813948    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 13:54:13.887264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.887230    4429 generic.go:356] "Generic (PLEG): container finished" podID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerID="f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd" exitCode=0
Oct 30 13:54:13.887483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.887308    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"
Oct 30 13:54:13.887483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.887325    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" event={"ID":"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b","Type":"ContainerDied","Data":"f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd"}
Oct 30 13:54:13.887483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.887427    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2" event={"ID":"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b","Type":"ContainerDied","Data":"db3c7c91cc6d28ff7e703ce717f0bf0c0b2d6dbbad2aa6ff14756b26a072d33b"}
Oct 30 13:54:13.887483 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.887454    4429 scope.go:122] "RemoveContainer" containerID="f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd"
Oct 30 13:54:13.901167 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.901140    4429 scope.go:122] "RemoveContainer" containerID="f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd"
Oct 30 13:54:13.901500 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:54:13.901478    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd\": container with ID starting with f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd not found: ID does not exist" containerID="f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd"
Oct 30 13:54:13.901601 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.901514    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd"} err="failed to get container status \"f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd\": rpc error: code = NotFound desc = could not find container \"f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd\": container with ID starting with f20f1a2f7f4a725693d02a87f2c7c1ba4718f8900066910f011ccc878e6867dd not found: ID does not exist"
Oct 30 13:54:13.954711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.954682    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-kube-api-access-blvp2\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-kube-api-access-blvp2\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.954882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.954747    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-credential-socket\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.954882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.954786    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-envoy\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.954882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.954810    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istiod-ca-cert\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.954882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.954853    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-socket\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.954882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.954875    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-podinfo\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.955172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.954968    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-token\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-token\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.955172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955015    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-certs\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.955172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955053    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-data\") pod \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\" (UID: \"ebcbb19a-e0e3-46d6-adcf-405ca61eb41b\") "
Oct 30 13:54:13.955172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955135    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-credential-socket" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:13.955461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955178    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-socket" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:13.955461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955216    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istiod-ca-cert" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:13.955461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955371    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-certs" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:13.955821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955473    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-data" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:13.955821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955744    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:13.955821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955767    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:13.955821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955783    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:13.955821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955800    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:13.955821 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.955814    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:13.957167 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.957140    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-kube-api-access-blvp2" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "kube-api-access-blvp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:13.957294 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.957179    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-envoy" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:13.957420 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.957404    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-token" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:13.957523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:13.957509    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-podinfo" pod "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" (UID: "ebcbb19a-e0e3-46d6-adcf-405ca61eb41b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:54:14.057181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:14.057145    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvp2\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-kube-api-access-blvp2\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:14.057181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:14.057179    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:14.057181 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:14.057188    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:14.057465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:14.057197    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:14.233950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:14.233911    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"]
Oct 30 13:54:14.247171 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:14.247144    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ff0e6-openshift-default-69954f5dd9-w26r2"]
Oct 30 13:54:16.076475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:16.076440    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" path="/var/lib/kubelet/pods/ebcbb19a-e0e3-46d6-adcf-405ca61eb41b/volumes"
Oct 30 13:54:17.976390 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:17.976351    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w" podUID="d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:54:21.788376 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.788342    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:54:21.950126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.950087    4429 generic.go:356] "Generic (PLEG): container finished" podID="d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" containerID="5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf" exitCode=0
Oct 30 13:54:21.950372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.950158    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"
Oct 30 13:54:21.950372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.950184    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w" event={"ID":"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a","Type":"ContainerDied","Data":"5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf"}
Oct 30 13:54:21.950372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.950278    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w" event={"ID":"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a","Type":"ContainerDied","Data":"53e0ae8d28e187343d52283c50e74564a848414d06f906e16f428de7c9d06a19"}
Oct 30 13:54:21.950372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.950340    4429 scope.go:122] "RemoveContainer" containerID="5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf"
Oct 30 13:54:21.962161 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.962136    4429 scope.go:122] "RemoveContainer" containerID="5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf"
Oct 30 13:54:21.962504 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:54:21.962480    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf\": container with ID starting with 5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf not found: ID does not exist" containerID="5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf"
Oct 30 13:54:21.962589 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.962513    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf"} err="failed to get container status \"5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf\": rpc error: code = NotFound desc = could not find container \"5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf\": container with ID starting with 5873e792f270e8f996269d80e6400b568e8760dfb435dba3cd52bbcb1595eddf not found: ID does not exist"
Oct 30 13:54:21.962731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.962714    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-kube-api-access-4lv65\" (UniqueName: \"kubernetes.io/projected/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-kube-api-access-4lv65\") pod \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\" (UID: \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\") "
Oct 30 13:54:21.962790 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.962775    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-plugins-config-volume\") pod \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\" (UID: \"d1f1e8a2-eb2a-4e11-8923-22a34e177c4a\") "
Oct 30 13:54:21.963080 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.963061    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-plugins-config-volume" pod "d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" (UID: "d1f1e8a2-eb2a-4e11-8923-22a34e177c4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:21.963395 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.963377    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:21.965025 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:21.965002    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-kube-api-access-4lv65" pod "d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" (UID: "d1f1e8a2-eb2a-4e11-8923-22a34e177c4a"). InnerVolumeSpecName "kube-api-access-4lv65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:22.013377 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.013270    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"]
Oct 30 13:54:22.013574 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.013527    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9" podUID="eb361f7e-e214-47b7-bc86-e37708877f24" containerName="maas-api" containerID="cri-o://1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13" gracePeriod=30
Oct 30 13:54:22.064547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.064495    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lv65\" (UniqueName: \"kubernetes.io/projected/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a-kube-api-access-4lv65\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:22.298888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.298850    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"]
Oct 30 13:54:22.304834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.304798    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-88e9f3-6f4f779dbd-c4w2w"]
Oct 30 13:54:22.325012 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.324985    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:54:22.472099 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.472062    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb361f7e-e214-47b7-bc86-e37708877f24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb361f7e-e214-47b7-bc86-e37708877f24-openshift-service-ca\") pod \"eb361f7e-e214-47b7-bc86-e37708877f24\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") "
Oct 30 13:54:22.472332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.472172    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb361f7e-e214-47b7-bc86-e37708877f24-kube-api-access-2bxkt\" (UniqueName: \"kubernetes.io/projected/eb361f7e-e214-47b7-bc86-e37708877f24-kube-api-access-2bxkt\") pod \"eb361f7e-e214-47b7-bc86-e37708877f24\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") "
Oct 30 13:54:22.472332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.472204    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-maas-api-tls\") pod \"eb361f7e-e214-47b7-bc86-e37708877f24\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") "
Oct 30 13:54:22.472332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.472246    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-metrics-certs\") pod \"eb361f7e-e214-47b7-bc86-e37708877f24\" (UID: \"eb361f7e-e214-47b7-bc86-e37708877f24\") "
Oct 30 13:54:22.472577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.472555    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb361f7e-e214-47b7-bc86-e37708877f24-openshift-service-ca" pod "eb361f7e-e214-47b7-bc86-e37708877f24" (UID: "eb361f7e-e214-47b7-bc86-e37708877f24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:22.472798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.472774    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb361f7e-e214-47b7-bc86-e37708877f24-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:22.474591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.474561    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-metrics-certs" pod "eb361f7e-e214-47b7-bc86-e37708877f24" (UID: "eb361f7e-e214-47b7-bc86-e37708877f24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:22.474698 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.474589    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-maas-api-tls" pod "eb361f7e-e214-47b7-bc86-e37708877f24" (UID: "eb361f7e-e214-47b7-bc86-e37708877f24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:22.474762 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.474741    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb361f7e-e214-47b7-bc86-e37708877f24-kube-api-access-2bxkt" pod "eb361f7e-e214-47b7-bc86-e37708877f24" (UID: "eb361f7e-e214-47b7-bc86-e37708877f24"). InnerVolumeSpecName "kube-api-access-2bxkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:22.574501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.574415    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxkt\" (UniqueName: \"kubernetes.io/projected/eb361f7e-e214-47b7-bc86-e37708877f24-kube-api-access-2bxkt\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:22.574501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.574457    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:22.574501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.574472    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb361f7e-e214-47b7-bc86-e37708877f24-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:22.960734 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.960696    4429 generic.go:356] "Generic (PLEG): container finished" podID="eb361f7e-e214-47b7-bc86-e37708877f24" containerID="1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13" exitCode=0
Oct 30 13:54:22.961216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.960736    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"
Oct 30 13:54:22.961216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.960774    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9" event={"ID":"eb361f7e-e214-47b7-bc86-e37708877f24","Type":"ContainerDied","Data":"1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13"}
Oct 30 13:54:22.961216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.960843    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9" event={"ID":"eb361f7e-e214-47b7-bc86-e37708877f24","Type":"ContainerDied","Data":"df29d89e83144be5f0323eb19a1f1c365c78276a722ec674905005b031de2ff1"}
Oct 30 13:54:22.961216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.960873    4429 scope.go:122] "RemoveContainer" containerID="1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13"
Oct 30 13:54:22.975377 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.975354    4429 scope.go:122] "RemoveContainer" containerID="1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13"
Oct 30 13:54:22.975714 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:54:22.975691    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13\": container with ID starting with 1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13 not found: ID does not exist" containerID="1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13"
Oct 30 13:54:22.975836 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:22.975724    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13"} err="failed to get container status \"1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13\": rpc error: code = NotFound desc = could not find container \"1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13\": container with ID starting with 1a6f81a23cd64de42e776daefa8dea08d1397f8b524809cf3d155290e68a4b13 not found: ID does not exist"
Oct 30 13:54:23.019659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:23.019629    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"]
Oct 30 13:54:23.032484 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:23.032455    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-63499e-7ccbd8b4cc-mjjm9"]
Oct 30 13:54:24.077041 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:24.077006    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" path="/var/lib/kubelet/pods/d1f1e8a2-eb2a-4e11-8923-22a34e177c4a/volumes"
Oct 30 13:54:24.077558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:24.077509    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb361f7e-e214-47b7-bc86-e37708877f24" path="/var/lib/kubelet/pods/eb361f7e-e214-47b7-bc86-e37708877f24/volumes"
Oct 30 13:54:34.057742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:34.057704    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"]
Oct 30 13:54:41.662680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:41.662635    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"]
Oct 30 13:54:42.377871 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:54:42.377829    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:54:49.079221 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.079177    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng" podUID="171bd558-2e74-4d2c-ad34-41118c9c26ef" containerName="main" containerID="cri-o://c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1" gracePeriod=45
Oct 30 13:54:49.365105 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.365078    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:54:49.532695 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.532660    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171bd558-2e74-4d2c-ad34-41118c9c26ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/171bd558-2e74-4d2c-ad34-41118c9c26ef-tls-certs\") pod \"171bd558-2e74-4d2c-ad34-41118c9c26ef\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") "
Oct 30 13:54:49.532884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.532729    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-dshm\") pod \"171bd558-2e74-4d2c-ad34-41118c9c26ef\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") "
Oct 30 13:54:49.532884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.532762    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-tmp-dir\") pod \"171bd558-2e74-4d2c-ad34-41118c9c26ef\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") "
Oct 30 13:54:49.532884 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.532793    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171bd558-2e74-4d2c-ad34-41118c9c26ef-kube-api-access-z8bhd\" (UniqueName: \"kubernetes.io/projected/171bd558-2e74-4d2c-ad34-41118c9c26ef-kube-api-access-z8bhd\") pod \"171bd558-2e74-4d2c-ad34-41118c9c26ef\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") "
Oct 30 13:54:49.533069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.532831    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-model-cache\") pod \"171bd558-2e74-4d2c-ad34-41118c9c26ef\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") "
Oct 30 13:54:49.533069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.532983    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-home\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-home\") pod \"171bd558-2e74-4d2c-ad34-41118c9c26ef\" (UID: \"171bd558-2e74-4d2c-ad34-41118c9c26ef\") "
Oct 30 13:54:49.533220 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.533175    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-tmp-dir" pod "171bd558-2e74-4d2c-ad34-41118c9c26ef" (UID: "171bd558-2e74-4d2c-ad34-41118c9c26ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:49.533333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.533269    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-model-cache" pod "171bd558-2e74-4d2c-ad34-41118c9c26ef" (UID: "171bd558-2e74-4d2c-ad34-41118c9c26ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:49.533333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.533309    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-home" pod "171bd558-2e74-4d2c-ad34-41118c9c26ef" (UID: "171bd558-2e74-4d2c-ad34-41118c9c26ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:49.533640 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.533620    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:49.533720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.533646    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:49.533720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.533658    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:49.535086 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.535061    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171bd558-2e74-4d2c-ad34-41118c9c26ef-tls-certs" pod "171bd558-2e74-4d2c-ad34-41118c9c26ef" (UID: "171bd558-2e74-4d2c-ad34-41118c9c26ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:49.535233 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.535212    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-dshm" pod "171bd558-2e74-4d2c-ad34-41118c9c26ef" (UID: "171bd558-2e74-4d2c-ad34-41118c9c26ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:49.535343 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.535251    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171bd558-2e74-4d2c-ad34-41118c9c26ef-kube-api-access-z8bhd" pod "171bd558-2e74-4d2c-ad34-41118c9c26ef" (UID: "171bd558-2e74-4d2c-ad34-41118c9c26ef"). InnerVolumeSpecName "kube-api-access-z8bhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:49.635066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.634967    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8bhd\" (UniqueName: \"kubernetes.io/projected/171bd558-2e74-4d2c-ad34-41118c9c26ef-kube-api-access-z8bhd\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:49.635066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.635007    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/171bd558-2e74-4d2c-ad34-41118c9c26ef-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:49.635066 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:49.635021    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/171bd558-2e74-4d2c-ad34-41118c9c26ef-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:50.172016 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.171984    4429 generic.go:356] "Generic (PLEG): container finished" podID="171bd558-2e74-4d2c-ad34-41118c9c26ef" containerID="c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1" exitCode=0
Oct 30 13:54:50.172512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.172031    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"
Oct 30 13:54:50.172512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.172073    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng" event={"ID":"171bd558-2e74-4d2c-ad34-41118c9c26ef","Type":"ContainerDied","Data":"c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1"}
Oct 30 13:54:50.172512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.172139    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng" event={"ID":"171bd558-2e74-4d2c-ad34-41118c9c26ef","Type":"ContainerDied","Data":"b82dc2d6e43fc758c9bab5c317da94cd718afdf2a8a94c4daf1cf4ad63eac90f"}
Oct 30 13:54:50.172512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.172158    4429 scope.go:122] "RemoveContainer" containerID="c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1"
Oct 30 13:54:50.182091 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.182075    4429 scope.go:122] "RemoveContainer" containerID="c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1"
Oct 30 13:54:50.182374 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:54:50.182353    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1\": container with ID starting with c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1 not found: ID does not exist" containerID="c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1"
Oct 30 13:54:50.182459 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.182389    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1"} err="failed to get container status \"c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1\": rpc error: code = NotFound desc = could not find container \"c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1\": container with ID starting with c60506638d615f3bed31d3a593527c5987f8f15a48c52b105ed0d52eef391dd1 not found: ID does not exist"
Oct 30 13:54:50.228585 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.228559    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"]
Oct 30 13:54:50.242171 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.242146    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng"]
Oct 30 13:54:50.264294 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:50.264251    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-517a25/rate-test-model-517a25-kserve-5795466769-rssng" podUID="171bd558-2e74-4d2c-ad34-41118c9c26ef" containerName="main" probeResult="failure" output="Get \"https://10.133.1.1:8000/ready\": context deadline exceeded"
Oct 30 13:54:52.076523 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:52.076485    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171bd558-2e74-4d2c-ad34-41118c9c26ef" path="/var/lib/kubelet/pods/171bd558-2e74-4d2c-ad34-41118c9c26ef/volumes"
Oct 30 13:54:52.457920 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:52.457884    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"]
Oct 30 13:54:52.493388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:52.493345    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"]
Oct 30 13:54:56.684654 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:56.684612    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6" podUID="eba1899f-c0d5-4484-aa5b-eb878b13b571" containerName="main" containerID="cri-o://3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e" gracePeriod=45
Oct 30 13:54:56.964614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:56.964585    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:54:57.028394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.028362    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-tmp-dir\") pod \"eba1899f-c0d5-4484-aa5b-eb878b13b571\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") "
Oct 30 13:54:57.028605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.028403    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-home\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-home\") pod \"eba1899f-c0d5-4484-aa5b-eb878b13b571\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") "
Oct 30 13:54:57.028605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.028446    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-model-cache\") pod \"eba1899f-c0d5-4484-aa5b-eb878b13b571\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") "
Oct 30 13:54:57.028605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.028485    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba1899f-c0d5-4484-aa5b-eb878b13b571-kube-api-access-7hc2c\" (UniqueName: \"kubernetes.io/projected/eba1899f-c0d5-4484-aa5b-eb878b13b571-kube-api-access-7hc2c\") pod \"eba1899f-c0d5-4484-aa5b-eb878b13b571\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") "
Oct 30 13:54:57.028605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.028586    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-dshm\") pod \"eba1899f-c0d5-4484-aa5b-eb878b13b571\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") "
Oct 30 13:54:57.029421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.028716    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eba1899f-c0d5-4484-aa5b-eb878b13b571-tls-certs\" (UniqueName: \"kubernetes.io/secret/eba1899f-c0d5-4484-aa5b-eb878b13b571-tls-certs\") pod \"eba1899f-c0d5-4484-aa5b-eb878b13b571\" (UID: \"eba1899f-c0d5-4484-aa5b-eb878b13b571\") "
Oct 30 13:54:57.029421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.028728    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-home" pod "eba1899f-c0d5-4484-aa5b-eb878b13b571" (UID: "eba1899f-c0d5-4484-aa5b-eb878b13b571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:57.029421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.028741    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-model-cache" pod "eba1899f-c0d5-4484-aa5b-eb878b13b571" (UID: "eba1899f-c0d5-4484-aa5b-eb878b13b571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:57.029421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.029224    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-tmp-dir" pod "eba1899f-c0d5-4484-aa5b-eb878b13b571" (UID: "eba1899f-c0d5-4484-aa5b-eb878b13b571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:57.029421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.029329    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:57.029421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.029348    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:57.029421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.029361    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:57.031635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.031612    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-dshm" pod "eba1899f-c0d5-4484-aa5b-eb878b13b571" (UID: "eba1899f-c0d5-4484-aa5b-eb878b13b571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:57.032021 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.032000    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba1899f-c0d5-4484-aa5b-eb878b13b571-tls-certs" pod "eba1899f-c0d5-4484-aa5b-eb878b13b571" (UID: "eba1899f-c0d5-4484-aa5b-eb878b13b571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:57.032335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.032317    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba1899f-c0d5-4484-aa5b-eb878b13b571-kube-api-access-7hc2c" pod "eba1899f-c0d5-4484-aa5b-eb878b13b571" (UID: "eba1899f-c0d5-4484-aa5b-eb878b13b571"). InnerVolumeSpecName "kube-api-access-7hc2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:57.131011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.130978    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hc2c\" (UniqueName: \"kubernetes.io/projected/eba1899f-c0d5-4484-aa5b-eb878b13b571-kube-api-access-7hc2c\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:57.131247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.131103    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba1899f-c0d5-4484-aa5b-eb878b13b571-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:57.131247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.131137    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba1899f-c0d5-4484-aa5b-eb878b13b571-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:57.224637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.224547    4429 generic.go:356] "Generic (PLEG): container finished" podID="eba1899f-c0d5-4484-aa5b-eb878b13b571" containerID="3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e" exitCode=0
Oct 30 13:54:57.224637 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.224591    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"
Oct 30 13:54:57.224900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.224636    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6" event={"ID":"eba1899f-c0d5-4484-aa5b-eb878b13b571","Type":"ContainerDied","Data":"3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e"}
Oct 30 13:54:57.224900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.224683    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6" event={"ID":"eba1899f-c0d5-4484-aa5b-eb878b13b571","Type":"ContainerDied","Data":"c14e9b3bae3a6279b0287ae3ed055657f775ed8819864374af93b14170a4d146"}
Oct 30 13:54:57.224900 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.224701    4429 scope.go:122] "RemoveContainer" containerID="3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e"
Oct 30 13:54:57.236558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.236538    4429 scope.go:122] "RemoveContainer" containerID="3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e"
Oct 30 13:54:57.236844 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:54:57.236823    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e\": container with ID starting with 3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e not found: ID does not exist" containerID="3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e"
Oct 30 13:54:57.236933 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.236851    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e"} err="failed to get container status \"3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e\": rpc error: code = NotFound desc = could not find container \"3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e\": container with ID starting with 3d98572ca1133950e6dc3191e4849199977a79887255b90423fad6f57504847e not found: ID does not exist"
Oct 30 13:54:57.306330 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.306295    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"]
Oct 30 13:54:57.313012 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.312985    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-241ffa/rate-test-model-241ffa-kserve-6549564bc7-lhgw6"]
Oct 30 13:54:57.458382 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.458344    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4" podUID="1ca25a94-48bd-4048-9955-09b137fbefe4" containerName="payload-processing" containerID="cri-o://06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc" gracePeriod=25
Oct 30 13:54:57.494624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:57.494529    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl" podUID="c74aba3a-269e-4d9c-88a3-4b224d852c36" containerName="payload-pre-processing" containerID="cri-o://0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94" gracePeriod=25
Oct 30 13:54:58.075526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:58.075491    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba1899f-c0d5-4484-aa5b-eb878b13b571" path="/var/lib/kubelet/pods/eba1899f-c0d5-4484-aa5b-eb878b13b571/volumes"
Oct 30 13:54:58.867525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:58.867491    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"]
Oct 30 13:54:58.867777 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:58.867751    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw" podUID="9017cc61-e4f0-403a-90f9-a0bebe754590" containerName="maas-api" containerID="cri-o://0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d" gracePeriod=30
Oct 30 13:54:59.151200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.151171    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:54:59.243045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.243007    4429 generic.go:356] "Generic (PLEG): container finished" podID="9017cc61-e4f0-403a-90f9-a0bebe754590" containerID="0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d" exitCode=0
Oct 30 13:54:59.243218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.243057    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"
Oct 30 13:54:59.243218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.243072    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw" event={"ID":"9017cc61-e4f0-403a-90f9-a0bebe754590","Type":"ContainerDied","Data":"0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d"}
Oct 30 13:54:59.243218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.243128    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw" event={"ID":"9017cc61-e4f0-403a-90f9-a0bebe754590","Type":"ContainerDied","Data":"a88aca376b4515fbe3fa97c3fdc851e7735302e744215e4d8f59606a3886e94d"}
Oct 30 13:54:59.243218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.243163    4429 scope.go:122] "RemoveContainer" containerID="0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d"
Oct 30 13:54:59.253647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.253616    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl" podUID="c74aba3a-269e-4d9c-88a3-4b224d852c36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:54:59.253882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.253627    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4" podUID="1ca25a94-48bd-4048-9955-09b137fbefe4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:54:59.254461 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.254442    4429 scope.go:122] "RemoveContainer" containerID="0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d"
Oct 30 13:54:59.254757 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:54:59.254738    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d\": container with ID starting with 0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d not found: ID does not exist" containerID="0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d"
Oct 30 13:54:59.254829 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.254763    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d"} err="failed to get container status \"0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d\": rpc error: code = NotFound desc = could not find container \"0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d\": container with ID starting with 0afebe136561ab65085d2d14a99caba43aa4ce44d287644fd9c396c513d7bd2d not found: ID does not exist"
Oct 30 13:54:59.257267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.257242    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9017cc61-e4f0-403a-90f9-a0bebe754590-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017cc61-e4f0-403a-90f9-a0bebe754590-openshift-service-ca\") pod \"9017cc61-e4f0-403a-90f9-a0bebe754590\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") "
Oct 30 13:54:59.257432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.257310    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9017cc61-e4f0-403a-90f9-a0bebe754590-kube-api-access-lm92n\" (UniqueName: \"kubernetes.io/projected/9017cc61-e4f0-403a-90f9-a0bebe754590-kube-api-access-lm92n\") pod \"9017cc61-e4f0-403a-90f9-a0bebe754590\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") "
Oct 30 13:54:59.257432 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.257346    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-metrics-certs\") pod \"9017cc61-e4f0-403a-90f9-a0bebe754590\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") "
Oct 30 13:54:59.257567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.257449    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-maas-api-tls\") pod \"9017cc61-e4f0-403a-90f9-a0bebe754590\" (UID: \"9017cc61-e4f0-403a-90f9-a0bebe754590\") "
Oct 30 13:54:59.258051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.258024    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9017cc61-e4f0-403a-90f9-a0bebe754590-openshift-service-ca" pod "9017cc61-e4f0-403a-90f9-a0bebe754590" (UID: "9017cc61-e4f0-403a-90f9-a0bebe754590"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:59.258176 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.258155    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017cc61-e4f0-403a-90f9-a0bebe754590-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:59.260469 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.260447    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9017cc61-e4f0-403a-90f9-a0bebe754590-kube-api-access-lm92n" pod "9017cc61-e4f0-403a-90f9-a0bebe754590" (UID: "9017cc61-e4f0-403a-90f9-a0bebe754590"). InnerVolumeSpecName "kube-api-access-lm92n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:59.260638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.260621    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-maas-api-tls" pod "9017cc61-e4f0-403a-90f9-a0bebe754590" (UID: "9017cc61-e4f0-403a-90f9-a0bebe754590"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:59.260688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.260648    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-metrics-certs" pod "9017cc61-e4f0-403a-90f9-a0bebe754590" (UID: "9017cc61-e4f0-403a-90f9-a0bebe754590"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:59.359378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.359337    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm92n\" (UniqueName: \"kubernetes.io/projected/9017cc61-e4f0-403a-90f9-a0bebe754590-kube-api-access-lm92n\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:59.359378 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.359375    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:59.359602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.359390    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017cc61-e4f0-403a-90f9-a0bebe754590-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:54:59.597250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.597218    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"]
Oct 30 13:54:59.609570 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:54:59.609529    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8740d4-786fd4f497-m2fcw"]
Oct 30 13:55:00.076209 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:00.076175    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9017cc61-e4f0-403a-90f9-a0bebe754590" path="/var/lib/kubelet/pods/9017cc61-e4f0-403a-90f9-a0bebe754590/volumes"
Oct 30 13:55:02.653184 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:02.653148    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"]
Oct 30 13:55:02.653870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:02.653389    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy" containerID="cri-o://1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1" gracePeriod=30
Oct 30 13:55:02.666832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:02.666801    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"]
Oct 30 13:55:02.667070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:02.667043    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy" containerID="cri-o://752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f" gracePeriod=30
Oct 30 13:55:07.942833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.942805    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:55:07.971332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971272    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e886c2e-e044-4f32-af32-9945c531b70f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e886c2e-e044-4f32-af32-9945c531b70f-istio-podinfo\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.971475 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971371    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-socket\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.971563 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971512    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-envoy\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.971626 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971586    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-credential-socket\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.971686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971621    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e886c2e-e044-4f32-af32-9945c531b70f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e886c2e-e044-4f32-af32-9945c531b70f-istiod-ca-cert\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.971686 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971647    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-socket" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:07.971793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971702    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-certs\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.971793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971733    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-kube-api-access-j88f8\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-kube-api-access-j88f8\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.971902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971815    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-data\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.971902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971847    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-credential-socket" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:07.971902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.971852    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-istio-token\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-istio-token\") pod \"4e886c2e-e044-4f32-af32-9945c531b70f\" (UID: \"4e886c2e-e044-4f32-af32-9945c531b70f\") "
Oct 30 13:55:07.972478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.972235    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-data" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:07.972478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.972249    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e886c2e-e044-4f32-af32-9945c531b70f-istiod-ca-cert" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:07.972478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.972326    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-certs" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:07.972718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.972494    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:07.972718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.972517    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:07.972718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.972531    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e886c2e-e044-4f32-af32-9945c531b70f-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:07.972718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.972547    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:07.972718 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.972559    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:07.974271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.974240    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-istio-token" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:07.974406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.974268    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-envoy" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:07.974406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.974335    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e886c2e-e044-4f32-af32-9945c531b70f-istio-podinfo" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:07.974498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.974434    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-kube-api-access-j88f8" pod "4e886c2e-e044-4f32-af32-9945c531b70f" (UID: "4e886c2e-e044-4f32-af32-9945c531b70f"). InnerVolumeSpecName "kube-api-access-j88f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:07.991496 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:07.991477    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:55:08.073447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073420    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-credential-socket\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.073651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073492    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-podinfo\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.073651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073518    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istiod-ca-cert\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.073651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073568    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-envoy\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.073651 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073625    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-token\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-token\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.073937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073660    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-kube-api-access-dq9zg\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-kube-api-access-dq9zg\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.073937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073744    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-data\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.073937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073810    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-credential-socket" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:08.073937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073857    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-certs\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.073937 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.073890    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-socket\") pod \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\" (UID: \"e64ff59a-c76d-41ab-9c2a-55c717e626cf\") "
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074385    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-certs" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074703    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-socket" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074785    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-data" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074832    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e886c2e-e044-4f32-af32-9945c531b70f-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074856    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j88f8\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-kube-api-access-j88f8\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074871    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074887    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074900    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e886c2e-e044-4f32-af32-9945c531b70f-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074906    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istiod-ca-cert" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074913    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e886c2e-e044-4f32-af32-9945c531b70f-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.075124 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.074938    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.077451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.077431    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-kube-api-access-dq9zg" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "kube-api-access-dq9zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:08.083138 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.083100    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-envoy" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:08.083758 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.083734    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-podinfo" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:08.084010 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.083982    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-token" pod "e64ff59a-c76d-41ab-9c2a-55c717e626cf" (UID: "e64ff59a-c76d-41ab-9c2a-55c717e626cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:08.176389 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.176354    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.176389 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.176389    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.176591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.176402    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.176591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.176414    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.176591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.176427    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9zg\" (UniqueName: \"kubernetes.io/projected/e64ff59a-c76d-41ab-9c2a-55c717e626cf-kube-api-access-dq9zg\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.176591 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.176439    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64ff59a-c76d-41ab-9c2a-55c717e626cf-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:08.306721 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.306631    4429 generic.go:356] "Generic (PLEG): container finished" podID="4e886c2e-e044-4f32-af32-9945c531b70f" containerID="1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1" exitCode=0
Oct 30 13:55:08.306721 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.306687    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"
Oct 30 13:55:08.306946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.306715    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" event={"ID":"4e886c2e-e044-4f32-af32-9945c531b70f","Type":"ContainerDied","Data":"1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1"}
Oct 30 13:55:08.306946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.306770    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg" event={"ID":"4e886c2e-e044-4f32-af32-9945c531b70f","Type":"ContainerDied","Data":"7d3ea754d8cda14fd4b098e0c63552dd37524919cf619504ce6759d5477517b4"}
Oct 30 13:55:08.306946 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.306793    4429 scope.go:122] "RemoveContainer" containerID="1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1"
Oct 30 13:55:08.308422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.308389    4429 generic.go:356] "Generic (PLEG): container finished" podID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerID="752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f" exitCode=0
Oct 30 13:55:08.308539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.308427    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" event={"ID":"e64ff59a-c76d-41ab-9c2a-55c717e626cf","Type":"ContainerDied","Data":"752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f"}
Oct 30 13:55:08.308539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.308464    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"
Oct 30 13:55:08.308539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.308481    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd" event={"ID":"e64ff59a-c76d-41ab-9c2a-55c717e626cf","Type":"ContainerDied","Data":"9f0c21363e9dcfc9c4657329046b9e42d25945a72f0f04ccfc73443f38f250f8"}
Oct 30 13:55:08.317632 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.317617    4429 scope.go:122] "RemoveContainer" containerID="1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1"
Oct 30 13:55:08.317916 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:08.317899    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1\": container with ID starting with 1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1 not found: ID does not exist" containerID="1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1"
Oct 30 13:55:08.317990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.317922    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1"} err="failed to get container status \"1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1\": rpc error: code = NotFound desc = could not find container \"1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1\": container with ID starting with 1e02aed75c8ac244e4a9109e9258662e8efafdb8e29c99fa0c13fe5b81838cf1 not found: ID does not exist"
Oct 30 13:55:08.317990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.317938    4429 scope.go:122] "RemoveContainer" containerID="752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f"
Oct 30 13:55:08.328239 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.328222    4429 scope.go:122] "RemoveContainer" containerID="752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f"
Oct 30 13:55:08.328531 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:08.328511    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f\": container with ID starting with 752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f not found: ID does not exist" containerID="752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f"
Oct 30 13:55:08.328595 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.328538    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f"} err="failed to get container status \"752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f\": rpc error: code = NotFound desc = could not find container \"752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f\": container with ID starting with 752f4e74e25e8e8e0096a23f61428c50245456109e791ca6a496ae111ad5b32f not found: ID does not exist"
Oct 30 13:55:08.365845 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.365813    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"]
Oct 30 13:55:08.379478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.379450    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7hbnd"]
Oct 30 13:55:08.420116 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.420080    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"]
Oct 30 13:55:08.422820 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:08.422799    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-xpssg"]
Oct 30 13:55:09.253447 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:09.253408    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4" podUID="1ca25a94-48bd-4048-9955-09b137fbefe4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:09.253864 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:09.253769    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl" podUID="c74aba3a-269e-4d9c-88a3-4b224d852c36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:10.088148 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:10.088100    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" path="/var/lib/kubelet/pods/4e886c2e-e044-4f32-af32-9945c531b70f/volumes"
Oct 30 13:55:10.088743 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:10.088723    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" path="/var/lib/kubelet/pods/e64ff59a-c76d-41ab-9c2a-55c717e626cf/volumes"
Oct 30 13:55:10.871888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:10.871855    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"]
Oct 30 13:55:10.872397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:10.872104    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy" containerID="cri-o://5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764" gracePeriod=30
Oct 30 13:55:12.372450 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:12.372412    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:55:16.139517 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.139461    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:55:16.284603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284572    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-kube-api-access-wgc75\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-kube-api-access-wgc75\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.284814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284629    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-certs\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.284814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284665    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-credential-socket\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.284814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284716    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b327a616-e506-492f-88b7-66e4143df16a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327a616-e506-492f-88b7-66e4143df16a-istio-podinfo\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.284814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284751    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-envoy\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.284814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284774    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-data\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.284814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284814    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-istio-token\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-istio-token\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.285159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284866    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-socket\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.285159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.284953    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b327a616-e506-492f-88b7-66e4143df16a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327a616-e506-492f-88b7-66e4143df16a-istiod-ca-cert\") pod \"b327a616-e506-492f-88b7-66e4143df16a\" (UID: \"b327a616-e506-492f-88b7-66e4143df16a\") "
Oct 30 13:55:16.285159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285018    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-credential-socket" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:16.285159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285034    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-certs" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:16.285159 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285112    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-data" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:16.285647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285245    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-socket" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:16.285647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285443    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b327a616-e506-492f-88b7-66e4143df16a-istiod-ca-cert" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:16.285647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285559    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.285647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285576    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b327a616-e506-492f-88b7-66e4143df16a-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.285647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285585    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.285647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285593    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.285647 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.285601    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.287148 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.287119    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-kube-api-access-wgc75" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "kube-api-access-wgc75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:16.287271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.287171    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b327a616-e506-492f-88b7-66e4143df16a-istio-podinfo" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:16.287557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.287535    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-envoy" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:16.287557 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.287548    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-istio-token" pod "b327a616-e506-492f-88b7-66e4143df16a" (UID: "b327a616-e506-492f-88b7-66e4143df16a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:16.367435 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.367406    4429 generic.go:356] "Generic (PLEG): container finished" podID="b327a616-e506-492f-88b7-66e4143df16a" containerID="5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764" exitCode=0
Oct 30 13:55:16.367577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.367456    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"
Oct 30 13:55:16.367577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.367496    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" event={"ID":"b327a616-e506-492f-88b7-66e4143df16a","Type":"ContainerDied","Data":"5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764"}
Oct 30 13:55:16.367577 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.367558    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh" event={"ID":"b327a616-e506-492f-88b7-66e4143df16a","Type":"ContainerDied","Data":"453d07283ff71918020635bbdb7173ff8df6822b4defc30befd2d883c12c5887"}
Oct 30 13:55:16.367755 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.367587    4429 scope.go:122] "RemoveContainer" containerID="5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764"
Oct 30 13:55:16.379041 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.379025    4429 scope.go:122] "RemoveContainer" containerID="63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6"
Oct 30 13:55:16.386679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.386658    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b327a616-e506-492f-88b7-66e4143df16a-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.386679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.386682    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b327a616-e506-492f-88b7-66e4143df16a-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.386679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.386690    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.386842 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.386699    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgc75\" (UniqueName: \"kubernetes.io/projected/b327a616-e506-492f-88b7-66e4143df16a-kube-api-access-wgc75\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:16.388412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.388395    4429 scope.go:122] "RemoveContainer" containerID="5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764"
Oct 30 13:55:16.388655 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:16.388638    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764\": container with ID starting with 5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764 not found: ID does not exist" containerID="5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764"
Oct 30 13:55:16.388703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.388664    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764"} err="failed to get container status \"5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764\": rpc error: code = NotFound desc = could not find container \"5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764\": container with ID starting with 5404471e671b75a7d1aea538960221d4b05ff64ee6c4b63b87f6e9a2d7eac764 not found: ID does not exist"
Oct 30 13:55:16.388703 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.388680    4429 scope.go:122] "RemoveContainer" containerID="63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6"
Oct 30 13:55:16.388939 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:16.388918    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6\": container with ID starting with 63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6 not found: ID does not exist" containerID="63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6"
Oct 30 13:55:16.388987 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.388945    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6"} err="failed to get container status \"63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6\": rpc error: code = NotFound desc = could not find container \"63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6\": container with ID starting with 63cc4376b3431327d5799f9cc3724c871fa0603848120e7576db76067a1b72a6 not found: ID does not exist"
Oct 30 13:55:16.427840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.427781    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"]
Oct 30 13:55:16.440970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:16.440950    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7a8b38-openshift-default-6b889d6d97-zxpfh"]
Oct 30 13:55:17.804422 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.804400    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:55:17.819263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.819245    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:55:17.902488 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.902454    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74aba3a-269e-4d9c-88a3-4b224d852c36-kube-api-access-8cwh5\" (UniqueName: \"kubernetes.io/projected/c74aba3a-269e-4d9c-88a3-4b224d852c36-kube-api-access-8cwh5\") pod \"c74aba3a-269e-4d9c-88a3-4b224d852c36\" (UID: \"c74aba3a-269e-4d9c-88a3-4b224d852c36\") "
Oct 30 13:55:17.902697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.902614    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca25a94-48bd-4048-9955-09b137fbefe4-kube-api-access-dmfjw\" (UniqueName: \"kubernetes.io/projected/1ca25a94-48bd-4048-9955-09b137fbefe4-kube-api-access-dmfjw\") pod \"1ca25a94-48bd-4048-9955-09b137fbefe4\" (UID: \"1ca25a94-48bd-4048-9955-09b137fbefe4\") "
Oct 30 13:55:17.902697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.902657    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74aba3a-269e-4d9c-88a3-4b224d852c36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74aba3a-269e-4d9c-88a3-4b224d852c36-plugins-config-volume\") pod \"c74aba3a-269e-4d9c-88a3-4b224d852c36\" (UID: \"c74aba3a-269e-4d9c-88a3-4b224d852c36\") "
Oct 30 13:55:17.902881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.902860    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca25a94-48bd-4048-9955-09b137fbefe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca25a94-48bd-4048-9955-09b137fbefe4-plugins-config-volume\") pod \"1ca25a94-48bd-4048-9955-09b137fbefe4\" (UID: \"1ca25a94-48bd-4048-9955-09b137fbefe4\") "
Oct 30 13:55:17.903117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.903092    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74aba3a-269e-4d9c-88a3-4b224d852c36-plugins-config-volume" pod "c74aba3a-269e-4d9c-88a3-4b224d852c36" (UID: "c74aba3a-269e-4d9c-88a3-4b224d852c36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:17.903231 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.903209    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca25a94-48bd-4048-9955-09b137fbefe4-plugins-config-volume" pod "1ca25a94-48bd-4048-9955-09b137fbefe4" (UID: "1ca25a94-48bd-4048-9955-09b137fbefe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:17.903832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.903808    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca25a94-48bd-4048-9955-09b137fbefe4-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:17.903924 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.903838    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c74aba3a-269e-4d9c-88a3-4b224d852c36-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:17.905188 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.905160    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74aba3a-269e-4d9c-88a3-4b224d852c36-kube-api-access-8cwh5" pod "c74aba3a-269e-4d9c-88a3-4b224d852c36" (UID: "c74aba3a-269e-4d9c-88a3-4b224d852c36"). InnerVolumeSpecName "kube-api-access-8cwh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:17.905388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:17.905369    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca25a94-48bd-4048-9955-09b137fbefe4-kube-api-access-dmfjw" pod "1ca25a94-48bd-4048-9955-09b137fbefe4" (UID: "1ca25a94-48bd-4048-9955-09b137fbefe4"). InnerVolumeSpecName "kube-api-access-dmfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:18.004509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.004452    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmfjw\" (UniqueName: \"kubernetes.io/projected/1ca25a94-48bd-4048-9955-09b137fbefe4-kube-api-access-dmfjw\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.004509 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.004474    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cwh5\" (UniqueName: \"kubernetes.io/projected/c74aba3a-269e-4d9c-88a3-4b224d852c36-kube-api-access-8cwh5\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:18.075892 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.075865    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b327a616-e506-492f-88b7-66e4143df16a" path="/var/lib/kubelet/pods/b327a616-e506-492f-88b7-66e4143df16a/volumes"
Oct 30 13:55:18.384032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.383947    4429 generic.go:356] "Generic (PLEG): container finished" podID="1ca25a94-48bd-4048-9955-09b137fbefe4" containerID="06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc" exitCode=0
Oct 30 13:55:18.384032 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.383999    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"
Oct 30 13:55:18.384304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.384038    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4" event={"ID":"1ca25a94-48bd-4048-9955-09b137fbefe4","Type":"ContainerDied","Data":"06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc"}
Oct 30 13:55:18.384304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.384104    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4" event={"ID":"1ca25a94-48bd-4048-9955-09b137fbefe4","Type":"ContainerDied","Data":"7c2f3f0172485487f21528c5db856f4112f70eca835c7f49e8bd56a8928cbcf4"}
Oct 30 13:55:18.384304 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.384132    4429 scope.go:122] "RemoveContainer" containerID="06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc"
Oct 30 13:55:18.385489 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.385466    4429 generic.go:356] "Generic (PLEG): container finished" podID="c74aba3a-269e-4d9c-88a3-4b224d852c36" containerID="0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94" exitCode=0
Oct 30 13:55:18.385587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.385506    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"
Oct 30 13:55:18.385587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.385508    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl" event={"ID":"c74aba3a-269e-4d9c-88a3-4b224d852c36","Type":"ContainerDied","Data":"0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94"}
Oct 30 13:55:18.385701 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.385603    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl" event={"ID":"c74aba3a-269e-4d9c-88a3-4b224d852c36","Type":"ContainerDied","Data":"9d401768ecf25215e33d7ae39d5dcf02bfcea088c37e3eec23bdafa2da22c4e6"}
Oct 30 13:55:18.394899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.394882    4429 scope.go:122] "RemoveContainer" containerID="06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc"
Oct 30 13:55:18.395181 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:18.395162    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc\": container with ID starting with 06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc not found: ID does not exist" containerID="06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc"
Oct 30 13:55:18.395222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.395190    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc"} err="failed to get container status \"06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc\": rpc error: code = NotFound desc = could not find container \"06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc\": container with ID starting with 06815f97ab3644526b804ee241de5d6f0929ff3a042b1038a93d8f5488d739fc not found: ID does not exist"
Oct 30 13:55:18.395222 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.395206    4429 scope.go:122] "RemoveContainer" containerID="0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94"
Oct 30 13:55:18.404054 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.404021    4429 scope.go:122] "RemoveContainer" containerID="0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94"
Oct 30 13:55:18.404294 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:18.404262    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94\": container with ID starting with 0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94 not found: ID does not exist" containerID="0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94"
Oct 30 13:55:18.404337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.404311    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94"} err="failed to get container status \"0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94\": rpc error: code = NotFound desc = could not find container \"0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94\": container with ID starting with 0928817d7679ff41cc614ea22f1fe5951bced35a93689e8a712fb81690bbba94 not found: ID does not exist"
Oct 30 13:55:18.443143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.443116    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"]
Oct 30 13:55:18.456921 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.456900    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1957dc-6774bd598b-xb4k4"]
Oct 30 13:55:18.484889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.484868    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"]
Oct 30 13:55:18.490435 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:18.490415    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1957dc-559d4d7687-m5qkl"]
Oct 30 13:55:19.935034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:19.935003    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"]
Oct 30 13:55:20.079230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:20.079195    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca25a94-48bd-4048-9955-09b137fbefe4" path="/var/lib/kubelet/pods/1ca25a94-48bd-4048-9955-09b137fbefe4/volumes"
Oct 30 13:55:20.079677 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:20.079660    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74aba3a-269e-4d9c-88a3-4b224d852c36" path="/var/lib/kubelet/pods/c74aba3a-269e-4d9c-88a3-4b224d852c36/volumes"
Oct 30 13:55:21.970749 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:21.970711    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"]
Oct 30 13:55:24.260669 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:24.260627    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"]
Oct 30 13:55:24.935733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:24.935693    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6" podUID="97305157-a41d-465e-8e94-1aa9dc38a166" containerName="payload-processing" containerID="cri-o://110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3" gracePeriod=25
Oct 30 13:55:26.971532 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:26.971467    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69" podUID="591c870e-000d-4d4d-a1a6-1f52be1a23da" containerName="payload-pre-processing" containerID="cri-o://37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769" gracePeriod=25
Oct 30 13:55:29.261635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:29.261593    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb" podUID="b3da904d-79b0-4f10-b122-fb0720de4008" containerName="payload-processing" containerID="cri-o://5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2" gracePeriod=25
Oct 30 13:55:32.011466 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:32.011430    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb" podUID="b3da904d-79b0-4f10-b122-fb0720de4008" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:33.448571 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:33.448468    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6" podUID="97305157-a41d-465e-8e94-1aa9dc38a166" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:36.729371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:36.729329    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69" podUID="591c870e-000d-4d4d-a1a6-1f52be1a23da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:38.695905 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:38.695874    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"]
Oct 30 13:55:39.167457 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.167379    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"]
Oct 30 13:55:39.167610 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.167562    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc" podUID="edfc88d7-a686-4947-af56-ff979b8d603d" containerName="maas-api" containerID="cri-o://d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60" gracePeriod=30
Oct 30 13:55:39.438547 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.438516    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:55:39.519017 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.518982    4429 generic.go:356] "Generic (PLEG): container finished" podID="edfc88d7-a686-4947-af56-ff979b8d603d" containerID="d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60" exitCode=0
Oct 30 13:55:39.519236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.519036    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"
Oct 30 13:55:39.519236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.519061    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc" event={"ID":"edfc88d7-a686-4947-af56-ff979b8d603d","Type":"ContainerDied","Data":"d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60"}
Oct 30 13:55:39.519236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.519128    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc" event={"ID":"edfc88d7-a686-4947-af56-ff979b8d603d","Type":"ContainerDied","Data":"ba523398c947454333ee46ddc11d3479e7fdabff4381b0132b5a40c5277be94e"}
Oct 30 13:55:39.519236 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.519156    4429 scope.go:122] "RemoveContainer" containerID="d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60"
Oct 30 13:55:39.529571 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.529543    4429 scope.go:122] "RemoveContainer" containerID="d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60"
Oct 30 13:55:39.529923 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:39.529894    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60\": container with ID starting with d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60 not found: ID does not exist" containerID="d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60"
Oct 30 13:55:39.530033 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.529932    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60"} err="failed to get container status \"d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60\": rpc error: code = NotFound desc = could not find container \"d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60\": container with ID starting with d74e9bb562453be4532343da0de310ef26237e75bf5564b40a92ee454902ce60 not found: ID does not exist"
Oct 30 13:55:39.549757 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.549722    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edfc88d7-a686-4947-af56-ff979b8d603d-kube-api-access-wj2tt\" (UniqueName: \"kubernetes.io/projected/edfc88d7-a686-4947-af56-ff979b8d603d-kube-api-access-wj2tt\") pod \"edfc88d7-a686-4947-af56-ff979b8d603d\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") "
Oct 30 13:55:39.549941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.549821    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-metrics-certs\") pod \"edfc88d7-a686-4947-af56-ff979b8d603d\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") "
Oct 30 13:55:39.549941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.549854    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls\") pod \"edfc88d7-a686-4947-af56-ff979b8d603d\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") "
Oct 30 13:55:39.549941 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.549887    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edfc88d7-a686-4947-af56-ff979b8d603d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfc88d7-a686-4947-af56-ff979b8d603d-openshift-service-ca\") pod \"edfc88d7-a686-4947-af56-ff979b8d603d\" (UID: \"edfc88d7-a686-4947-af56-ff979b8d603d\") "
Oct 30 13:55:39.550337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.550268    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edfc88d7-a686-4947-af56-ff979b8d603d-openshift-service-ca" pod "edfc88d7-a686-4947-af56-ff979b8d603d" (UID: "edfc88d7-a686-4947-af56-ff979b8d603d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:39.552108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.552080    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-metrics-certs" pod "edfc88d7-a686-4947-af56-ff979b8d603d" (UID: "edfc88d7-a686-4947-af56-ff979b8d603d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:39.552202 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.552146    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls" pod "edfc88d7-a686-4947-af56-ff979b8d603d" (UID: "edfc88d7-a686-4947-af56-ff979b8d603d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:39.552246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.552194    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edfc88d7-a686-4947-af56-ff979b8d603d-kube-api-access-wj2tt" pod "edfc88d7-a686-4947-af56-ff979b8d603d" (UID: "edfc88d7-a686-4947-af56-ff979b8d603d"). InnerVolumeSpecName "kube-api-access-wj2tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:39.651705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.651671    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:39.651705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.651706    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edfc88d7-a686-4947-af56-ff979b8d603d-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:39.651919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.651720    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj2tt\" (UniqueName: \"kubernetes.io/projected/edfc88d7-a686-4947-af56-ff979b8d603d-kube-api-access-wj2tt\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:39.651919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.651732    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edfc88d7-a686-4947-af56-ff979b8d603d-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:39.870464 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.870423    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"]
Oct 30 13:55:39.883951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:39.883916    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e9cb4a-7fd75c88c-tg5zc"]
Oct 30 13:55:40.077048 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:40.077012    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edfc88d7-a686-4947-af56-ff979b8d603d" path="/var/lib/kubelet/pods/edfc88d7-a686-4947-af56-ff979b8d603d/volumes"
Oct 30 13:55:41.777421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:41.777385    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"]
Oct 30 13:55:41.778024 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:41.777649    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy" containerID="cri-o://565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4" gracePeriod=30
Oct 30 13:55:42.011262 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:42.011229    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb" podUID="b3da904d-79b0-4f10-b122-fb0720de4008" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:42.372927 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:42.372894    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:55:43.447782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:43.447749    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6" podUID="97305157-a41d-465e-8e94-1aa9dc38a166" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:45.213098 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.213074    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:55:45.324628 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.324599    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97305157-a41d-465e-8e94-1aa9dc38a166-kube-api-access-jz799\" (UniqueName: \"kubernetes.io/projected/97305157-a41d-465e-8e94-1aa9dc38a166-kube-api-access-jz799\") pod \"97305157-a41d-465e-8e94-1aa9dc38a166\" (UID: \"97305157-a41d-465e-8e94-1aa9dc38a166\") "
Oct 30 13:55:45.324831 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.324719    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97305157-a41d-465e-8e94-1aa9dc38a166-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97305157-a41d-465e-8e94-1aa9dc38a166-plugins-config-volume\") pod \"97305157-a41d-465e-8e94-1aa9dc38a166\" (UID: \"97305157-a41d-465e-8e94-1aa9dc38a166\") "
Oct 30 13:55:45.325162 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.325125    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97305157-a41d-465e-8e94-1aa9dc38a166-plugins-config-volume" pod "97305157-a41d-465e-8e94-1aa9dc38a166" (UID: "97305157-a41d-465e-8e94-1aa9dc38a166"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:45.325332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.325202    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97305157-a41d-465e-8e94-1aa9dc38a166-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:45.326957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.326928    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97305157-a41d-465e-8e94-1aa9dc38a166-kube-api-access-jz799" pod "97305157-a41d-465e-8e94-1aa9dc38a166" (UID: "97305157-a41d-465e-8e94-1aa9dc38a166"). InnerVolumeSpecName "kube-api-access-jz799". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:45.426716 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.426681    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz799\" (UniqueName: \"kubernetes.io/projected/97305157-a41d-465e-8e94-1aa9dc38a166-kube-api-access-jz799\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:45.560653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.560564    4429 generic.go:356] "Generic (PLEG): container finished" podID="97305157-a41d-465e-8e94-1aa9dc38a166" containerID="110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3" exitCode=0
Oct 30 13:55:45.560653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.560610    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"
Oct 30 13:55:45.560653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.560643    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6" event={"ID":"97305157-a41d-465e-8e94-1aa9dc38a166","Type":"ContainerDied","Data":"110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3"}
Oct 30 13:55:45.560925 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.560694    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6" event={"ID":"97305157-a41d-465e-8e94-1aa9dc38a166","Type":"ContainerDied","Data":"bad8cefccd74b7f32757cb3bdc29bef25bd3b6dc306e81506408e1359107848c"}
Oct 30 13:55:45.560925 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.560717    4429 scope.go:122] "RemoveContainer" containerID="110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3"
Oct 30 13:55:45.570936 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.570916    4429 scope.go:122] "RemoveContainer" containerID="110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3"
Oct 30 13:55:45.571203 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:45.571177    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3\": container with ID starting with 110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3 not found: ID does not exist" containerID="110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3"
Oct 30 13:55:45.571349 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.571211    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3"} err="failed to get container status \"110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3\": rpc error: code = NotFound desc = could not find container \"110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3\": container with ID starting with 110ab71cde5e68b3a595a09bf916305b0d1794bec07bbc4685fe8c7b1fc883b3 not found: ID does not exist"
Oct 30 13:55:45.617952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.617913    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"]
Oct 30 13:55:45.622697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:45.622659    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e9cb4a-64dff45dc6-vvrg6"]
Oct 30 13:55:46.075463 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:46.075420    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97305157-a41d-465e-8e94-1aa9dc38a166" path="/var/lib/kubelet/pods/97305157-a41d-465e-8e94-1aa9dc38a166/volumes"
Oct 30 13:55:46.729804 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:46.729758    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69" podUID="591c870e-000d-4d4d-a1a6-1f52be1a23da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:55:47.107255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.107226    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:55:47.236656 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.236631    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:55:47.268827 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.268655    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-kube-api-access-9kqsk\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-kube-api-access-9kqsk\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.268827 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.268741    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-data\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.268827 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.268795    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-podinfo\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.268827 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.268826    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-socket\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.269204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.268916    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6206414c-fa76-42c9-a13e-7d4a45fea76b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6206414c-fa76-42c9-a13e-7d4a45fea76b-istiod-ca-cert\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.269204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.268970    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-envoy\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.269204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.269054    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-credential-socket\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.269204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.269124    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-certs\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.269204 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.269193    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-token\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-token\") pod \"6206414c-fa76-42c9-a13e-7d4a45fea76b\" (UID: \"6206414c-fa76-42c9-a13e-7d4a45fea76b\") "
Oct 30 13:55:47.269668 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.269615    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-socket" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.269761 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.269687    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-data" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.271050 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.270357    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6206414c-fa76-42c9-a13e-7d4a45fea76b-istiod-ca-cert" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:47.271050 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.270433    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-credential-socket" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.271050 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.270499    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-certs" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.272228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.272200    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-envoy" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:47.272638 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.272615    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-token" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:47.272873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.272854    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-podinfo" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:47.273352 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.273327    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-kube-api-access-9kqsk" pod "6206414c-fa76-42c9-a13e-7d4a45fea76b" (UID: "6206414c-fa76-42c9-a13e-7d4a45fea76b"). InnerVolumeSpecName "kube-api-access-9kqsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:47.308731 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.308695    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"]
Oct 30 13:55:47.331028 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.330999    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"]
Oct 30 13:55:47.370736 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.370697    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591c870e-000d-4d4d-a1a6-1f52be1a23da-kube-api-access-kjt5s\" (UniqueName: \"kubernetes.io/projected/591c870e-000d-4d4d-a1a6-1f52be1a23da-kube-api-access-kjt5s\") pod \"591c870e-000d-4d4d-a1a6-1f52be1a23da\" (UID: \"591c870e-000d-4d4d-a1a6-1f52be1a23da\") "
Oct 30 13:55:47.370928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.370858    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591c870e-000d-4d4d-a1a6-1f52be1a23da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c870e-000d-4d4d-a1a6-1f52be1a23da-plugins-config-volume\") pod \"591c870e-000d-4d4d-a1a6-1f52be1a23da\" (UID: \"591c870e-000d-4d4d-a1a6-1f52be1a23da\") "
Oct 30 13:55:47.371225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371200    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591c870e-000d-4d4d-a1a6-1f52be1a23da-plugins-config-volume" pod "591c870e-000d-4d4d-a1a6-1f52be1a23da" (UID: "591c870e-000d-4d4d-a1a6-1f52be1a23da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:47.371225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371211    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.371225 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371232    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.371625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371242    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6206414c-fa76-42c9-a13e-7d4a45fea76b-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.371625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371251    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.371625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371259    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.371625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371273    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.371625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371315    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.371625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371327    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqsk\" (UniqueName: \"kubernetes.io/projected/6206414c-fa76-42c9-a13e-7d4a45fea76b-kube-api-access-9kqsk\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.371625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.371340    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6206414c-fa76-42c9-a13e-7d4a45fea76b-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.373230 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.373206    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591c870e-000d-4d4d-a1a6-1f52be1a23da-kube-api-access-kjt5s" pod "591c870e-000d-4d4d-a1a6-1f52be1a23da" (UID: "591c870e-000d-4d4d-a1a6-1f52be1a23da"). InnerVolumeSpecName "kube-api-access-kjt5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:47.472160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.472124    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c870e-000d-4d4d-a1a6-1f52be1a23da-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.472160 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.472156    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjt5s\" (UniqueName: \"kubernetes.io/projected/591c870e-000d-4d4d-a1a6-1f52be1a23da-kube-api-access-kjt5s\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:47.574782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.574689    4429 generic.go:356] "Generic (PLEG): container finished" podID="591c870e-000d-4d4d-a1a6-1f52be1a23da" containerID="37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769" exitCode=0
Oct 30 13:55:47.574782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.574742    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"
Oct 30 13:55:47.574782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.574765    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69" event={"ID":"591c870e-000d-4d4d-a1a6-1f52be1a23da","Type":"ContainerDied","Data":"37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769"}
Oct 30 13:55:47.575085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.574819    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69" event={"ID":"591c870e-000d-4d4d-a1a6-1f52be1a23da","Type":"ContainerDied","Data":"4b7c5f261d8a0a380a11a1268c89af0eb8fda056c4da11b93b03a4592e2f3152"}
Oct 30 13:55:47.575085 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.574853    4429 scope.go:122] "RemoveContainer" containerID="37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769"
Oct 30 13:55:47.576859 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.576837    4429 generic.go:356] "Generic (PLEG): container finished" podID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerID="565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4" exitCode=0
Oct 30 13:55:47.576989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.576881    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"
Oct 30 13:55:47.576989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.576898    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" event={"ID":"6206414c-fa76-42c9-a13e-7d4a45fea76b","Type":"ContainerDied","Data":"565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4"}
Oct 30 13:55:47.576989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.576931    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6" event={"ID":"6206414c-fa76-42c9-a13e-7d4a45fea76b","Type":"ContainerDied","Data":"0718df1ee51d52faaba858e200c1dc7092ad9af319a4b6887585741250128ebf"}
Oct 30 13:55:47.587663 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.587645    4429 scope.go:122] "RemoveContainer" containerID="37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769"
Oct 30 13:55:47.588063 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:47.588042    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769\": container with ID starting with 37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769 not found: ID does not exist" containerID="37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769"
Oct 30 13:55:47.588166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.588073    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769"} err="failed to get container status \"37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769\": rpc error: code = NotFound desc = could not find container \"37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769\": container with ID starting with 37087a5e3d444d8cfc7c34773affb6a535b04da4fd7a7a18d21cb13261be5769 not found: ID does not exist"
Oct 30 13:55:47.588166 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.588095    4429 scope.go:122] "RemoveContainer" containerID="565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4"
Oct 30 13:55:47.600689 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.600657    4429 scope.go:122] "RemoveContainer" containerID="69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc"
Oct 30 13:55:47.611178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.611154    4429 scope.go:122] "RemoveContainer" containerID="565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4"
Oct 30 13:55:47.611625 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:47.611497    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4\": container with ID starting with 565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4 not found: ID does not exist" containerID="565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4"
Oct 30 13:55:47.611625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.611534    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4"} err="failed to get container status \"565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4\": rpc error: code = NotFound desc = could not find container \"565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4\": container with ID starting with 565107f3693390ab629f9da12fdbcd787bf211a49dc02bdb34361d3bab1eb3c4 not found: ID does not exist"
Oct 30 13:55:47.611625 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.611555    4429 scope.go:122] "RemoveContainer" containerID="69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc"
Oct 30 13:55:47.611860 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:47.611841    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc\": container with ID starting with 69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc not found: ID does not exist" containerID="69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc"
Oct 30 13:55:47.611899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.611867    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc"} err="failed to get container status \"69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc\": rpc error: code = NotFound desc = could not find container \"69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc\": container with ID starting with 69e7d9aa21de738b01d8ad05a77bbd43b910363e0c5c8d1427413e2b5ecee0dc not found: ID does not exist"
Oct 30 13:55:47.654470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.654341    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"]
Oct 30 13:55:47.657545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.657517    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-98dabb-bb4fcbb5f-s7l69"]
Oct 30 13:55:47.692265 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.692220    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"]
Oct 30 13:55:47.705423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:47.705392    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-98dabb-openshift-default-778885789c-62pl6"]
Oct 30 13:55:48.074978 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:48.074949    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591c870e-000d-4d4d-a1a6-1f52be1a23da" path="/var/lib/kubelet/pods/591c870e-000d-4d4d-a1a6-1f52be1a23da/volumes"
Oct 30 13:55:48.075433 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:48.075418    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" path="/var/lib/kubelet/pods/6206414c-fa76-42c9-a13e-7d4a45fea76b/volumes"
Oct 30 13:55:48.949131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:48.949095    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"]
Oct 30 13:55:48.949397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:48.949371    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy" containerID="cri-o://e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6" gracePeriod=30
Oct 30 13:55:49.521492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.521438    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:55:49.595751 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.595718    4429 generic.go:356] "Generic (PLEG): container finished" podID="b3da904d-79b0-4f10-b122-fb0720de4008" containerID="5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2" exitCode=0
Oct 30 13:55:49.595956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.595765    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"
Oct 30 13:55:49.595956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.595811    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb" event={"ID":"b3da904d-79b0-4f10-b122-fb0720de4008","Type":"ContainerDied","Data":"5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2"}
Oct 30 13:55:49.595956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.595877    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb" event={"ID":"b3da904d-79b0-4f10-b122-fb0720de4008","Type":"ContainerDied","Data":"77acaefff0d78a044bae8f9b7f42d186cce7f61ebd0cc25589bcb3d4486f9726"}
Oct 30 13:55:49.595956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.595911    4429 scope.go:122] "RemoveContainer" containerID="5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2"
Oct 30 13:55:49.606968 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.606940    4429 scope.go:122] "RemoveContainer" containerID="5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2"
Oct 30 13:55:49.608750 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:49.608718    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2\": container with ID starting with 5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2 not found: ID does not exist" containerID="5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2"
Oct 30 13:55:49.608870 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.608764    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2"} err="failed to get container status \"5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2\": rpc error: code = NotFound desc = could not find container \"5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2\": container with ID starting with 5487cae994e9d2618c94c1eb87112970d745c5398dd2981081a4af78940da9e2 not found: ID does not exist"
Oct 30 13:55:49.698027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.697994    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3da904d-79b0-4f10-b122-fb0720de4008-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3da904d-79b0-4f10-b122-fb0720de4008-plugins-config-volume\") pod \"b3da904d-79b0-4f10-b122-fb0720de4008\" (UID: \"b3da904d-79b0-4f10-b122-fb0720de4008\") "
Oct 30 13:55:49.698232 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.698086    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3da904d-79b0-4f10-b122-fb0720de4008-kube-api-access-n47k7\" (UniqueName: \"kubernetes.io/projected/b3da904d-79b0-4f10-b122-fb0720de4008-kube-api-access-n47k7\") pod \"b3da904d-79b0-4f10-b122-fb0720de4008\" (UID: \"b3da904d-79b0-4f10-b122-fb0720de4008\") "
Oct 30 13:55:49.698430 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.698407    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3da904d-79b0-4f10-b122-fb0720de4008-plugins-config-volume" pod "b3da904d-79b0-4f10-b122-fb0720de4008" (UID: "b3da904d-79b0-4f10-b122-fb0720de4008"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:49.700312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.700279    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da904d-79b0-4f10-b122-fb0720de4008-kube-api-access-n47k7" pod "b3da904d-79b0-4f10-b122-fb0720de4008" (UID: "b3da904d-79b0-4f10-b122-fb0720de4008"). InnerVolumeSpecName "kube-api-access-n47k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:49.799105 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.799025    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3da904d-79b0-4f10-b122-fb0720de4008-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:49.799105 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.799054    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47k7\" (UniqueName: \"kubernetes.io/projected/b3da904d-79b0-4f10-b122-fb0720de4008-kube-api-access-n47k7\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:49.866189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.866152    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"]
Oct 30 13:55:49.900943 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.900913    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"]
Oct 30 13:55:49.962336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.962302    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"]
Oct 30 13:55:49.967742 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:49.967706    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2d2a0b-67f88c66b4-7w5vb"]
Oct 30 13:55:50.076151 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:50.076069    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3da904d-79b0-4f10-b122-fb0720de4008" path="/var/lib/kubelet/pods/b3da904d-79b0-4f10-b122-fb0720de4008/volumes"
Oct 30 13:55:51.088580 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:51.088546    4429 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:55:51.089005 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:51.088611    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:55:53.103255 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:53.103212    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"]
Oct 30 13:55:53.140298 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:53.140248    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"]
Oct 30 13:55:53.717404 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:53.717360    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m" podUID="9830a470-d928-4baf-b5b0-da84535d7cd8" containerName="main" containerID="cri-o://cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e" gracePeriod=45
Oct 30 13:55:53.808268 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:53.808219    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m" podUID="9830a470-d928-4baf-b5b0-da84535d7cd8" containerName="main" probeResult="failure" output="Get \"https://10.133.1.3:8000/ready\": dial tcp 10.133.1.3:8000: connect: connection refused"
Oct 30 13:55:54.083155 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.083122    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:55:54.212903 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.212871    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:55:54.251135 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251025    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-tmp-dir\") pod \"9830a470-d928-4baf-b5b0-da84535d7cd8\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") "
Oct 30 13:55:54.251340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251177    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9830a470-d928-4baf-b5b0-da84535d7cd8-tls-certs\" (UniqueName: \"kubernetes.io/secret/9830a470-d928-4baf-b5b0-da84535d7cd8-tls-certs\") pod \"9830a470-d928-4baf-b5b0-da84535d7cd8\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") "
Oct 30 13:55:54.251340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251229    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-dshm\") pod \"9830a470-d928-4baf-b5b0-da84535d7cd8\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") "
Oct 30 13:55:54.251340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251314    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-model-cache\") pod \"9830a470-d928-4baf-b5b0-da84535d7cd8\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") "
Oct 30 13:55:54.251340 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251312    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-tmp-dir" pod "9830a470-d928-4baf-b5b0-da84535d7cd8" (UID: "9830a470-d928-4baf-b5b0-da84535d7cd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.251565 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251444    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9830a470-d928-4baf-b5b0-da84535d7cd8-kube-api-access-qw85v\" (UniqueName: \"kubernetes.io/projected/9830a470-d928-4baf-b5b0-da84535d7cd8-kube-api-access-qw85v\") pod \"9830a470-d928-4baf-b5b0-da84535d7cd8\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") "
Oct 30 13:55:54.251565 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251480    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-home\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-home\") pod \"9830a470-d928-4baf-b5b0-da84535d7cd8\" (UID: \"9830a470-d928-4baf-b5b0-da84535d7cd8\") "
Oct 30 13:55:54.251827 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251807    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-home" pod "9830a470-d928-4baf-b5b0-da84535d7cd8" (UID: "9830a470-d928-4baf-b5b0-da84535d7cd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.251923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.251836    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-model-cache" pod "9830a470-d928-4baf-b5b0-da84535d7cd8" (UID: "9830a470-d928-4baf-b5b0-da84535d7cd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.252218 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.252198    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.252315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.252224    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.252315 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.252238    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.253931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.253910    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9830a470-d928-4baf-b5b0-da84535d7cd8-tls-certs" pod "9830a470-d928-4baf-b5b0-da84535d7cd8" (UID: "9830a470-d928-4baf-b5b0-da84535d7cd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:55:54.254388 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.254362    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-dshm" pod "9830a470-d928-4baf-b5b0-da84535d7cd8" (UID: "9830a470-d928-4baf-b5b0-da84535d7cd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.254501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.254451    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9830a470-d928-4baf-b5b0-da84535d7cd8-kube-api-access-qw85v" pod "9830a470-d928-4baf-b5b0-da84535d7cd8" (UID: "9830a470-d928-4baf-b5b0-da84535d7cd8"). InnerVolumeSpecName "kube-api-access-qw85v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:54.352818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.352784    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-kube-api-access-wwdg7\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-kube-api-access-wwdg7\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.352836    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922d2e60-03b5-4ae7-b439-3b57d47d2902-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d2e60-03b5-4ae7-b439-3b57d47d2902-istiod-ca-cert\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.352971    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-envoy\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353052 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353036    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-socket\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353084    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-certs\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353115    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-data\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353149    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-credential-socket\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353247 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353237    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-podinfo\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353271    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-token\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-token\") pod \"922d2e60-03b5-4ae7-b439-3b57d47d2902\" (UID: \"922d2e60-03b5-4ae7-b439-3b57d47d2902\") "
Oct 30 13:55:54.353498 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353374    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922d2e60-03b5-4ae7-b439-3b57d47d2902-istiod-ca-cert" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:54.353719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353582    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-socket" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.353719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353611    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-data" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.353874 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353846    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.353940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353880    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.353940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353842    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-credential-socket" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.353940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353893    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-certs" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.353940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353897    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9830a470-d928-4baf-b5b0-da84535d7cd8-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.353940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353934    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9830a470-d928-4baf-b5b0-da84535d7cd8-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.354429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353950    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d2e60-03b5-4ae7-b439-3b57d47d2902-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.354429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.353976    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw85v\" (UniqueName: \"kubernetes.io/projected/9830a470-d928-4baf-b5b0-da84535d7cd8-kube-api-access-qw85v\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.355685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.355664    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-podinfo" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:54.355875 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.355858    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-token" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:54.355974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.355948    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-envoy" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.356089 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.356036    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-kube-api-access-wwdg7" pod "922d2e60-03b5-4ae7-b439-3b57d47d2902" (UID: "922d2e60-03b5-4ae7-b439-3b57d47d2902"). InnerVolumeSpecName "kube-api-access-wwdg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:54.454967 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.454926    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwdg7\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-kube-api-access-wwdg7\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.454967 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.454964    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.454967 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.454974    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.455240 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.454985    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d2e60-03b5-4ae7-b439-3b57d47d2902-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.455240 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.454995    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.455240 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.455003    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922d2e60-03b5-4ae7-b439-3b57d47d2902-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.629697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.629606    4429 generic.go:356] "Generic (PLEG): container finished" podID="9830a470-d928-4baf-b5b0-da84535d7cd8" containerID="cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e" exitCode=0
Oct 30 13:55:54.629697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.629655    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"
Oct 30 13:55:54.629939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.629711    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m" event={"ID":"9830a470-d928-4baf-b5b0-da84535d7cd8","Type":"ContainerDied","Data":"cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e"}
Oct 30 13:55:54.629939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.629761    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m" event={"ID":"9830a470-d928-4baf-b5b0-da84535d7cd8","Type":"ContainerDied","Data":"24cb51c9c57aad4ff387b230b5108048a7528a36b250a4776f931772a49501e4"}
Oct 30 13:55:54.629939 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.629780    4429 scope.go:122] "RemoveContainer" containerID="cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e"
Oct 30 13:55:54.631655 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.631632    4429 generic.go:356] "Generic (PLEG): container finished" podID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerID="e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6" exitCode=0
Oct 30 13:55:54.631784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.631729    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" event={"ID":"922d2e60-03b5-4ae7-b439-3b57d47d2902","Type":"ContainerDied","Data":"e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6"}
Oct 30 13:55:54.631784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.631770    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf" event={"ID":"922d2e60-03b5-4ae7-b439-3b57d47d2902","Type":"ContainerDied","Data":"b78c221fa995fb06a0823041fac04581e12c6948b64ae10dec623863ad7d81ea"}
Oct 30 13:55:54.631883 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.631821    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"
Oct 30 13:55:54.641800 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.641708    4429 scope.go:122] "RemoveContainer" containerID="cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e"
Oct 30 13:55:54.642103 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:54.642075    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e\": container with ID starting with cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e not found: ID does not exist" containerID="cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e"
Oct 30 13:55:54.642172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.642120    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e"} err="failed to get container status \"cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e\": rpc error: code = NotFound desc = could not find container \"cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e\": container with ID starting with cacfcf81860db89029084d01570cc53ca4d753cc3d49befc437e68343e07879e not found: ID does not exist"
Oct 30 13:55:54.642172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.642148    4429 scope.go:122] "RemoveContainer" containerID="e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6"
Oct 30 13:55:54.651961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.651943    4429 scope.go:122] "RemoveContainer" containerID="eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f"
Oct 30 13:55:54.665567 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.665549    4429 scope.go:122] "RemoveContainer" containerID="e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6"
Oct 30 13:55:54.665967 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:54.665936    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6\": container with ID starting with e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6 not found: ID does not exist" containerID="e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6"
Oct 30 13:55:54.666068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.665982    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6"} err="failed to get container status \"e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6\": rpc error: code = NotFound desc = could not find container \"e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6\": container with ID starting with e8548608327cc5381a63762ae31ca8f0500c5a89824e5ce0a9e21996a7e02af6 not found: ID does not exist"
Oct 30 13:55:54.666068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.666004    4429 scope.go:122] "RemoveContainer" containerID="eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f"
Oct 30 13:55:54.666348 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:55:54.666331    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f\": container with ID starting with eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f not found: ID does not exist" containerID="eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f"
Oct 30 13:55:54.666406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.666355    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f"} err="failed to get container status \"eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f\": rpc error: code = NotFound desc = could not find container \"eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f\": container with ID starting with eb2c0ab0d2171184fc37761d725a77537a30bc619769a4ade13586d01c25487f not found: ID does not exist"
Oct 30 13:55:54.687205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.687169    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"]
Oct 30 13:55:54.690889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.690855    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-451a21/rate-test-model-451a21-kserve-78b74dc557-2h79m"]
Oct 30 13:55:54.730973 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.730937    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"]
Oct 30 13:55:54.734434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.734406    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e9cb4a-openshift-default-77c74c484c-ptxrf"]
Oct 30 13:55:54.866501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.866456    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5" podUID="d75912f6-090d-4853-8c85-c714af76d1f4" containerName="payload-processing" containerID="cri-o://806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2" gracePeriod=25
Oct 30 13:55:54.901375 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:54.901243    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75" podUID="b0d701f4-05ec-48c8-898b-d2b954e8f368" containerName="payload-pre-processing" containerID="cri-o://46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c" gracePeriod=25
Oct 30 13:55:56.075956 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:56.075873    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" path="/var/lib/kubelet/pods/922d2e60-03b5-4ae7-b439-3b57d47d2902/volumes"
Oct 30 13:55:56.076416 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:56.076302    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9830a470-d928-4baf-b5b0-da84535d7cd8" path="/var/lib/kubelet/pods/9830a470-d928-4baf-b5b0-da84535d7cd8/volumes"
Oct 30 13:55:58.104435 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:58.104368    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc" podUID="9dfa2574-66ba-4023-b91f-37ed39e299e1" containerName="payload-processing" containerID="cri-o://4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33" gracePeriod=25
Oct 30 13:55:58.141069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:58.141024    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m" podUID="03680b2d-91b9-40e8-a3c5-5061188362b4" containerName="payload-pre-processing" containerID="cri-o://8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac" gracePeriod=25
Oct 30 13:55:59.912048 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:59.912009    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"]
Oct 30 13:55:59.947117 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:55:59.947081    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"]
Oct 30 13:56:00.679083 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:00.679047    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"]
Oct 30 13:56:00.713040 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:00.713008    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"]
Oct 30 13:56:01.682786 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:01.682752    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5" podUID="d75912f6-090d-4853-8c85-c714af76d1f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:01.683217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:01.682820    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75" podUID="b0d701f4-05ec-48c8-898b-d2b954e8f368" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:02.330709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.330668    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" containerID="cri-o://8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b" gracePeriod=45
Oct 30 13:56:02.356779 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.356722    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main" containerID="cri-o://c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b" gracePeriod=45
Oct 30 13:56:02.629380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.629351    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:56:02.639848 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.639825    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-home\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-home\") pod \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") "
Oct 30 13:56:02.639995 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.639902    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-model-cache\") pod \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") "
Oct 30 13:56:02.640064 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640050    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tmp-dir\") pod \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") "
Oct 30 13:56:02.640131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640082    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tls-certs\") pod \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") "
Oct 30 13:56:02.640131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640106    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-dshm\") pod \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") "
Oct 30 13:56:02.640250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640114    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-home" pod "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" (UID: "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.640250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640144    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-kube-api-access-7s8gq\" (UniqueName: \"kubernetes.io/projected/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-kube-api-access-7s8gq\") pod \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\" (UID: \"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e\") "
Oct 30 13:56:02.640250 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640212    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-model-cache" pod "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" (UID: "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.640444 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640321    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tmp-dir" pod "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" (UID: "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.640645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640621    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.640924 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640649    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.640924 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.640664    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.642685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.642583    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-dshm" pod "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" (UID: "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.642685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.642586    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tls-certs" pod "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" (UID: "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:02.642685 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.642659    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-kube-api-access-7s8gq" pod "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" (UID: "0d4a09b0-d45a-4c6b-9c22-80c35693bc8e"). InnerVolumeSpecName "kube-api-access-7s8gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:02.665346 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.665325    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:56:02.689451 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.689420    4429 generic.go:356] "Generic (PLEG): container finished" podID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerID="c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b" exitCode=0
Oct 30 13:56:02.689914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.689463    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"
Oct 30 13:56:02.689914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.689508    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" event={"ID":"f271de4a-9331-4b94-8d60-24a3eeb2df0c","Type":"ContainerDied","Data":"c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b"}
Oct 30 13:56:02.689914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.689548    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x" event={"ID":"f271de4a-9331-4b94-8d60-24a3eeb2df0c","Type":"ContainerDied","Data":"fe6260bce3f259ffa8e5ceb374139465403a0fc3c237181cda234c225b6b42a6"}
Oct 30 13:56:02.689914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.689584    4429 scope.go:122] "RemoveContainer" containerID="c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b"
Oct 30 13:56:02.691992 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.691964    4429 generic.go:356] "Generic (PLEG): container finished" podID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerID="8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b" exitCode=0
Oct 30 13:56:02.692146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.692127    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" event={"ID":"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e","Type":"ContainerDied","Data":"8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b"}
Oct 30 13:56:02.692223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.692163    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" event={"ID":"0d4a09b0-d45a-4c6b-9c22-80c35693bc8e","Type":"ContainerDied","Data":"bc69fcb883ff3c39aa4c8e86cf89e27b5e19a3e557158014834cc51bb641bdf0"}
Oct 30 13:56:02.692279 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.692220    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"
Oct 30 13:56:02.703966 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.703530    4429 scope.go:122] "RemoveContainer" containerID="c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b"
Oct 30 13:56:02.707125 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:02.707093    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b\": container with ID starting with c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b not found: ID does not exist" containerID="c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b"
Oct 30 13:56:02.707263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.707133    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b"} err="failed to get container status \"c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b\": rpc error: code = NotFound desc = could not find container \"c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b\": container with ID starting with c73b926ad13e1e9beae11e54e2f59024c7b1b7b5b786bd855429203884d7935b not found: ID does not exist"
Oct 30 13:56:02.707263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.707158    4429 scope.go:122] "RemoveContainer" containerID="8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b"
Oct 30 13:56:02.719121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.719094    4429 scope.go:122] "RemoveContainer" containerID="8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b"
Oct 30 13:56:02.719508 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:02.719481    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b\": container with ID starting with 8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b not found: ID does not exist" containerID="8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b"
Oct 30 13:56:02.719605 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.719519    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b"} err="failed to get container status \"8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b\": rpc error: code = NotFound desc = could not find container \"8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b\": container with ID starting with 8db6937be694f4a6592f7205957df12fded26d589eb7799c5db882df0f65488b not found: ID does not exist"
Oct 30 13:56:02.741147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.741107    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tls-certs\") pod \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") "
Oct 30 13:56:02.741349 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.741160    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-model-cache\") pod \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") "
Oct 30 13:56:02.741349 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.741269    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tmp-dir\") pod \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") "
Oct 30 13:56:02.741486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.741347    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-home\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-home\") pod \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") "
Oct 30 13:56:02.741486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.741383    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f271de4a-9331-4b94-8d60-24a3eeb2df0c-kube-api-access-chntf\" (UniqueName: \"kubernetes.io/projected/f271de4a-9331-4b94-8d60-24a3eeb2df0c-kube-api-access-chntf\") pod \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") "
Oct 30 13:56:02.741486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.741429    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-dshm\") pod \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\" (UID: \"f271de4a-9331-4b94-8d60-24a3eeb2df0c\") "
Oct 30 13:56:02.741659 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.741515    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-model-cache" pod "f271de4a-9331-4b94-8d60-24a3eeb2df0c" (UID: "f271de4a-9331-4b94-8d60-24a3eeb2df0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.741953 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.741928    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-home" pod "f271de4a-9331-4b94-8d60-24a3eeb2df0c" (UID: "f271de4a-9331-4b94-8d60-24a3eeb2df0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.742224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.742054    4429 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-model-cache\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.742224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.742074    4429 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-home\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.742224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.742088    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.742224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.742099    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.742224 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.742110    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8gq\" (UniqueName: \"kubernetes.io/projected/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e-kube-api-access-7s8gq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.742784 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.742304    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tmp-dir" pod "f271de4a-9331-4b94-8d60-24a3eeb2df0c" (UID: "f271de4a-9331-4b94-8d60-24a3eeb2df0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.744560 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.744536    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tls-certs" pod "f271de4a-9331-4b94-8d60-24a3eeb2df0c" (UID: "f271de4a-9331-4b94-8d60-24a3eeb2df0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:02.744661 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.744611    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-dshm" pod "f271de4a-9331-4b94-8d60-24a3eeb2df0c" (UID: "f271de4a-9331-4b94-8d60-24a3eeb2df0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:56:02.745833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.745809    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f271de4a-9331-4b94-8d60-24a3eeb2df0c-kube-api-access-chntf" pod "f271de4a-9331-4b94-8d60-24a3eeb2df0c" (UID: "f271de4a-9331-4b94-8d60-24a3eeb2df0c"). InnerVolumeSpecName "kube-api-access-chntf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:02.777646 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.777571    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"]
Oct 30 13:56:02.781137 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.781105    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm"]
Oct 30 13:56:02.843645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.843611    4429 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tls-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.843645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.843637    4429 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-tmp-dir\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.843645 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.843647    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chntf\" (UniqueName: \"kubernetes.io/projected/f271de4a-9331-4b94-8d60-24a3eeb2df0c-kube-api-access-chntf\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:02.843871 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:02.843656    4429 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f271de4a-9331-4b94-8d60-24a3eeb2df0c-dshm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:03.037581 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:03.037542    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"]
Oct 30 13:56:03.040948 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:03.040922    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e652ca/premium-w4-e652ca-kserve-79d85d4446-v547x"]
Oct 30 13:56:03.581927 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:03.581883    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e652ca/distinct-2-w4-e652ca-kserve-8b7b67dbb-c8bvm" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:56:04.075029 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:04.074993    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" path="/var/lib/kubelet/pods/0d4a09b0-d45a-4c6b-9c22-80c35693bc8e/volumes"
Oct 30 13:56:04.075450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:04.075425    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" path="/var/lib/kubelet/pods/f271de4a-9331-4b94-8d60-24a3eeb2df0c/volumes"
Oct 30 13:56:04.913238 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:04.913192    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52" podUID="b724c857-22c4-41c1-a1f4-85548435e237" containerName="payload-processing" containerID="cri-o://abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd" gracePeriod=25
Oct 30 13:56:04.947478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:04.947441    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx" podUID="bc7d3080-95f3-4fca-9680-24a2b982f3d0" containerName="payload-pre-processing" containerID="cri-o://3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6" gracePeriod=25
Oct 30 13:56:05.596592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:05.596555    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m" podUID="03680b2d-91b9-40e8-a3c5-5061188362b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:05.680335 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:05.680249    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq" podUID="cabcd6da-a086-4864-b2e7-0d41500b6c83" containerName="payload-processing" containerID="cri-o://a1060cb4fc582028ced69e3694438f8749e3b23a50a253b88e517c384ddce6ea" gracePeriod=25
Oct 30 13:56:05.714027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:05.713992    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp" podUID="33e2e18d-e039-43aa-8822-52590f759cfd" containerName="payload-pre-processing" containerID="cri-o://b328e2572e1bdcf54db55100113f21b92f0db76b63254a895aa7ed16e102f79e" gracePeriod=25
Oct 30 13:56:06.425301 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.421381    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"]
Oct 30 13:56:06.425301 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.421658    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts" podUID="bc24016f-ddfd-4c79-a015-4684b56b817f" containerName="maas-api" containerID="cri-o://249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6" gracePeriod=30
Oct 30 13:56:06.603576 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.603508    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc" podUID="9dfa2574-66ba-4023-b91f-37ed39e299e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:06.710694 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.710661    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:56:06.720196 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.720163    4429 generic.go:356] "Generic (PLEG): container finished" podID="bc24016f-ddfd-4c79-a015-4684b56b817f" containerID="249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6" exitCode=0
Oct 30 13:56:06.720384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.720230    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"
Oct 30 13:56:06.720384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.720248    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts" event={"ID":"bc24016f-ddfd-4c79-a015-4684b56b817f","Type":"ContainerDied","Data":"249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6"}
Oct 30 13:56:06.720384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.720340    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts" event={"ID":"bc24016f-ddfd-4c79-a015-4684b56b817f","Type":"ContainerDied","Data":"70f474e1abd9de04c93efcd0f011f47c70cae7416c224ba47efd82d83644eeff"}
Oct 30 13:56:06.720384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.720365    4429 scope.go:122] "RemoveContainer" containerID="249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6"
Oct 30 13:56:06.729843 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.729822    4429 scope.go:122] "RemoveContainer" containerID="249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6"
Oct 30 13:56:06.730109 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:06.730091    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6\": container with ID starting with 249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6 not found: ID does not exist" containerID="249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6"
Oct 30 13:56:06.730147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.730121    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6"} err="failed to get container status \"249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6\": rpc error: code = NotFound desc = could not find container \"249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6\": container with ID starting with 249b273faed83f792d8a3ca226ecb6119f83b2beab1be2caa908bc1eba7246a6 not found: ID does not exist"
Oct 30 13:56:06.787932 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.787902    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc24016f-ddfd-4c79-a015-4684b56b817f-kube-api-access-q5jpw\" (UniqueName: \"kubernetes.io/projected/bc24016f-ddfd-4c79-a015-4684b56b817f-kube-api-access-q5jpw\") pod \"bc24016f-ddfd-4c79-a015-4684b56b817f\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") "
Oct 30 13:56:06.788149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.788014    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-metrics-certs\") pod \"bc24016f-ddfd-4c79-a015-4684b56b817f\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") "
Oct 30 13:56:06.788149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.788047    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc24016f-ddfd-4c79-a015-4684b56b817f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc24016f-ddfd-4c79-a015-4684b56b817f-openshift-service-ca\") pod \"bc24016f-ddfd-4c79-a015-4684b56b817f\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") "
Oct 30 13:56:06.788149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.788094    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls\") pod \"bc24016f-ddfd-4c79-a015-4684b56b817f\" (UID: \"bc24016f-ddfd-4c79-a015-4684b56b817f\") "
Oct 30 13:56:06.790406 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.790371    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc24016f-ddfd-4c79-a015-4684b56b817f-openshift-service-ca" pod "bc24016f-ddfd-4c79-a015-4684b56b817f" (UID: "bc24016f-ddfd-4c79-a015-4684b56b817f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:06.791271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.791063    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-metrics-certs" pod "bc24016f-ddfd-4c79-a015-4684b56b817f" (UID: "bc24016f-ddfd-4c79-a015-4684b56b817f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:06.791271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.791083    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls" pod "bc24016f-ddfd-4c79-a015-4684b56b817f" (UID: "bc24016f-ddfd-4c79-a015-4684b56b817f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:06.791271 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.791232    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc24016f-ddfd-4c79-a015-4684b56b817f-kube-api-access-q5jpw" pod "bc24016f-ddfd-4c79-a015-4684b56b817f" (UID: "bc24016f-ddfd-4c79-a015-4684b56b817f"). InnerVolumeSpecName "kube-api-access-q5jpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:06.889681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.889648    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5jpw\" (UniqueName: \"kubernetes.io/projected/bc24016f-ddfd-4c79-a015-4684b56b817f-kube-api-access-q5jpw\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:06.889681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.889679    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:06.889681 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.889701    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc24016f-ddfd-4c79-a015-4684b56b817f-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:06.889935 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:06.889713    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc24016f-ddfd-4c79-a015-4684b56b817f-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:07.069316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:07.069248    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"]
Oct 30 13:56:07.075242 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:07.075211    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-92855c-56568b447d-97xts"]
Oct 30 13:56:08.076409 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:08.076370    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc24016f-ddfd-4c79-a015-4684b56b817f" path="/var/lib/kubelet/pods/bc24016f-ddfd-4c79-a015-4684b56b817f/volumes"
Oct 30 13:56:10.399891 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.399860    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"]
Oct 30 13:56:10.400440 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.400110    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8" podUID="6be5a43a-2320-4e63-821a-e15e67da2f81" containerName="maas-api" containerID="cri-o://843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f" gracePeriod=30
Oct 30 13:56:10.677446 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.677421    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:56:10.738199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.738158    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-maas-api-tls\") pod \"6be5a43a-2320-4e63-821a-e15e67da2f81\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") "
Oct 30 13:56:10.738578 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.738556    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-metrics-certs\") pod \"6be5a43a-2320-4e63-821a-e15e67da2f81\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") "
Oct 30 13:56:10.738923 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.738887    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be5a43a-2320-4e63-821a-e15e67da2f81-kube-api-access-25p2r\" (UniqueName: \"kubernetes.io/projected/6be5a43a-2320-4e63-821a-e15e67da2f81-kube-api-access-25p2r\") pod \"6be5a43a-2320-4e63-821a-e15e67da2f81\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") "
Oct 30 13:56:10.739037 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.738988    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be5a43a-2320-4e63-821a-e15e67da2f81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be5a43a-2320-4e63-821a-e15e67da2f81-openshift-service-ca\") pod \"6be5a43a-2320-4e63-821a-e15e67da2f81\" (UID: \"6be5a43a-2320-4e63-821a-e15e67da2f81\") "
Oct 30 13:56:10.739408 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.739377    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be5a43a-2320-4e63-821a-e15e67da2f81-openshift-service-ca" pod "6be5a43a-2320-4e63-821a-e15e67da2f81" (UID: "6be5a43a-2320-4e63-821a-e15e67da2f81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:10.739919 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.739900    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be5a43a-2320-4e63-821a-e15e67da2f81-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:10.742798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.742751    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-maas-api-tls" pod "6be5a43a-2320-4e63-821a-e15e67da2f81" (UID: "6be5a43a-2320-4e63-821a-e15e67da2f81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:10.744026 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.743998    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-metrics-certs" pod "6be5a43a-2320-4e63-821a-e15e67da2f81" (UID: "6be5a43a-2320-4e63-821a-e15e67da2f81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:10.745365 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.745337    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be5a43a-2320-4e63-821a-e15e67da2f81-kube-api-access-25p2r" pod "6be5a43a-2320-4e63-821a-e15e67da2f81" (UID: "6be5a43a-2320-4e63-821a-e15e67da2f81"). InnerVolumeSpecName "kube-api-access-25p2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:10.750781 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.750731    4429 generic.go:356] "Generic (PLEG): container finished" podID="6be5a43a-2320-4e63-821a-e15e67da2f81" containerID="843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f" exitCode=0
Oct 30 13:56:10.750950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.750812    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"
Oct 30 13:56:10.750950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.750820    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8" event={"ID":"6be5a43a-2320-4e63-821a-e15e67da2f81","Type":"ContainerDied","Data":"843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f"}
Oct 30 13:56:10.750950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.750899    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8" event={"ID":"6be5a43a-2320-4e63-821a-e15e67da2f81","Type":"ContainerDied","Data":"d0fbc3ef19abbdab09d8d38afc6a91b541049e7fa0bd3deaa7e4cabaee841700"}
Oct 30 13:56:10.751125 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.751010    4429 scope.go:122] "RemoveContainer" containerID="843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f"
Oct 30 13:56:10.772512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.772491    4429 scope.go:122] "RemoveContainer" containerID="843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f"
Oct 30 13:56:10.772856 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:10.772837    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f\": container with ID starting with 843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f not found: ID does not exist" containerID="843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f"
Oct 30 13:56:10.772915 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.772865    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f"} err="failed to get container status \"843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f\": rpc error: code = NotFound desc = could not find container \"843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f\": container with ID starting with 843dd3575340317b35a05f53f814ca274710149ff608549372d7252789b3736f not found: ID does not exist"
Oct 30 13:56:10.810339 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.809968    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"]
Oct 30 13:56:10.814013 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.813980    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-517a25-789948487d-9scf8"]
Oct 30 13:56:10.840811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.840771    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:10.840811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.840803    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25p2r\" (UniqueName: \"kubernetes.io/projected/6be5a43a-2320-4e63-821a-e15e67da2f81-kube-api-access-25p2r\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:10.840811 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:10.840814    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be5a43a-2320-4e63-821a-e15e67da2f81-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:11.680022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:11.679981    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52" podUID="b724c857-22c4-41c1-a1f4-85548435e237" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:11.680480 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:11.680087    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx" podUID="bc7d3080-95f3-4fca-9680-24a2b982f3d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:11.683197 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:11.683169    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75" podUID="b0d701f4-05ec-48c8-898b-d2b954e8f368" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:11.683333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:11.683194    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5" podUID="d75912f6-090d-4853-8c85-c714af76d1f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:12.074864 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:12.074832    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be5a43a-2320-4e63-821a-e15e67da2f81" path="/var/lib/kubelet/pods/6be5a43a-2320-4e63-821a-e15e67da2f81/volumes"
Oct 30 13:56:12.372333 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:12.372234    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:56:15.106697 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.103488    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq" podUID="cabcd6da-a086-4864-b2e7-0d41500b6c83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:15.107387 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.107312    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp" podUID="33e2e18d-e039-43aa-8822-52590f759cfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:15.179413 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.179387    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:56:15.209470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.209447    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:56:15.280688 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.280653    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"]
Oct 30 13:56:15.288446 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.288413    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d701f4-05ec-48c8-898b-d2b954e8f368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d701f4-05ec-48c8-898b-d2b954e8f368-plugins-config-volume\") pod \"b0d701f4-05ec-48c8-898b-d2b954e8f368\" (UID: \"b0d701f4-05ec-48c8-898b-d2b954e8f368\") "
Oct 30 13:56:15.288624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.288524    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75912f6-090d-4853-8c85-c714af76d1f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75912f6-090d-4853-8c85-c714af76d1f4-plugins-config-volume\") pod \"d75912f6-090d-4853-8c85-c714af76d1f4\" (UID: \"d75912f6-090d-4853-8c85-c714af76d1f4\") "
Oct 30 13:56:15.288624 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.288602    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75912f6-090d-4853-8c85-c714af76d1f4-kube-api-access-rs8nj\" (UniqueName: \"kubernetes.io/projected/d75912f6-090d-4853-8c85-c714af76d1f4-kube-api-access-rs8nj\") pod \"d75912f6-090d-4853-8c85-c714af76d1f4\" (UID: \"d75912f6-090d-4853-8c85-c714af76d1f4\") "
Oct 30 13:56:15.288733 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.288669    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d701f4-05ec-48c8-898b-d2b954e8f368-kube-api-access-c88kb\" (UniqueName: \"kubernetes.io/projected/b0d701f4-05ec-48c8-898b-d2b954e8f368-kube-api-access-c88kb\") pod \"b0d701f4-05ec-48c8-898b-d2b954e8f368\" (UID: \"b0d701f4-05ec-48c8-898b-d2b954e8f368\") "
Oct 30 13:56:15.288851 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.288823    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d701f4-05ec-48c8-898b-d2b954e8f368-plugins-config-volume" pod "b0d701f4-05ec-48c8-898b-d2b954e8f368" (UID: "b0d701f4-05ec-48c8-898b-d2b954e8f368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:15.288987 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.288963    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75912f6-090d-4853-8c85-c714af76d1f4-plugins-config-volume" pod "d75912f6-090d-4853-8c85-c714af76d1f4" (UID: "d75912f6-090d-4853-8c85-c714af76d1f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:15.289123 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.289107    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d701f4-05ec-48c8-898b-d2b954e8f368-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:15.289185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.289126    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d75912f6-090d-4853-8c85-c714af76d1f4-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:15.291199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.291171    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d701f4-05ec-48c8-898b-d2b954e8f368-kube-api-access-c88kb" pod "b0d701f4-05ec-48c8-898b-d2b954e8f368" (UID: "b0d701f4-05ec-48c8-898b-d2b954e8f368"). InnerVolumeSpecName "kube-api-access-c88kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:15.291434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.291414    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75912f6-090d-4853-8c85-c714af76d1f4-kube-api-access-rs8nj" pod "d75912f6-090d-4853-8c85-c714af76d1f4" (UID: "d75912f6-090d-4853-8c85-c714af76d1f4"). InnerVolumeSpecName "kube-api-access-rs8nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:15.390575 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.390488    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs8nj\" (UniqueName: \"kubernetes.io/projected/d75912f6-090d-4853-8c85-c714af76d1f4-kube-api-access-rs8nj\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:15.390575 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.390521    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c88kb\" (UniqueName: \"kubernetes.io/projected/b0d701f4-05ec-48c8-898b-d2b954e8f368-kube-api-access-c88kb\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:15.596969 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.596930    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m" podUID="03680b2d-91b9-40e8-a3c5-5061188362b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:15.781514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.781473    4429 generic.go:356] "Generic (PLEG): container finished" podID="b0d701f4-05ec-48c8-898b-d2b954e8f368" containerID="46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c" exitCode=0
Oct 30 13:56:15.781728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.781524    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"
Oct 30 13:56:15.781728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.781551    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75" event={"ID":"b0d701f4-05ec-48c8-898b-d2b954e8f368","Type":"ContainerDied","Data":"46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c"}
Oct 30 13:56:15.781728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.781608    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75" event={"ID":"b0d701f4-05ec-48c8-898b-d2b954e8f368","Type":"ContainerDied","Data":"e21cfbe7e724f67d8f84a69773a5a6ca92535c1e6f3d956658bea89da61c4747"}
Oct 30 13:56:15.781728 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.781640    4429 scope.go:122] "RemoveContainer" containerID="46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c"
Oct 30 13:56:15.783372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.783348    4429 generic.go:356] "Generic (PLEG): container finished" podID="d75912f6-090d-4853-8c85-c714af76d1f4" containerID="806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2" exitCode=0
Oct 30 13:56:15.783485 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.783412    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"
Oct 30 13:56:15.783485 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.783425    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5" event={"ID":"d75912f6-090d-4853-8c85-c714af76d1f4","Type":"ContainerDied","Data":"806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2"}
Oct 30 13:56:15.783485 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.783464    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5" event={"ID":"d75912f6-090d-4853-8c85-c714af76d1f4","Type":"ContainerDied","Data":"6dc7eeb9bd37b22fb71b7fd9e7aaec0dbb29908e13d91254c2e8e62b575d592c"}
Oct 30 13:56:15.792828 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.792810    4429 scope.go:122] "RemoveContainer" containerID="46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c"
Oct 30 13:56:15.793163 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:15.793139    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c\": container with ID starting with 46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c not found: ID does not exist" containerID="46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c"
Oct 30 13:56:15.793274 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.793172    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c"} err="failed to get container status \"46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c\": rpc error: code = NotFound desc = could not find container \"46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c\": container with ID starting with 46bf411f68e0efe73d792cdd4b45c603d8a983d4b0a3a196279bd24a35a8489c not found: ID does not exist"
Oct 30 13:56:15.793274 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.793193    4429 scope.go:122] "RemoveContainer" containerID="806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2"
Oct 30 13:56:15.802627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.802611    4429 scope.go:122] "RemoveContainer" containerID="806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2"
Oct 30 13:56:15.802913 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:15.802894    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2\": container with ID starting with 806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2 not found: ID does not exist" containerID="806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2"
Oct 30 13:56:15.802950 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.802921    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2"} err="failed to get container status \"806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2\": rpc error: code = NotFound desc = could not find container \"806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2\": container with ID starting with 806831138dfcd4f1fbe580059d916bc84ae21288fdeaf94e630a5d6b4ddbc8d2 not found: ID does not exist"
Oct 30 13:56:15.838513 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.838484    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"]
Oct 30 13:56:15.842928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.842884    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4db150-679bfc745b-h5cx5"]
Oct 30 13:56:15.881269 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.881224    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"]
Oct 30 13:56:15.885007 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:15.884979    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4db150-5f6d995d79-9dg75"]
Oct 30 13:56:16.074666 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:16.074588    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d701f4-05ec-48c8-898b-d2b954e8f368" path="/var/lib/kubelet/pods/b0d701f4-05ec-48c8-898b-d2b954e8f368/volumes"
Oct 30 13:56:16.075051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:16.075037    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75912f6-090d-4853-8c85-c714af76d1f4" path="/var/lib/kubelet/pods/d75912f6-090d-4853-8c85-c714af76d1f4/volumes"
Oct 30 13:56:16.601906 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:16.601865    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc" podUID="9dfa2574-66ba-4023-b91f-37ed39e299e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:18.436386 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.433039    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 13:56:18.475531 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.475503    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 13:56:18.529869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.529837    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dfa2574-66ba-4023-b91f-37ed39e299e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfa2574-66ba-4023-b91f-37ed39e299e1-plugins-config-volume\") pod \"9dfa2574-66ba-4023-b91f-37ed39e299e1\" (UID: \"9dfa2574-66ba-4023-b91f-37ed39e299e1\") "
Oct 30 13:56:18.530069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.529913    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfa2574-66ba-4023-b91f-37ed39e299e1-kube-api-access-42dgj\" (UniqueName: \"kubernetes.io/projected/9dfa2574-66ba-4023-b91f-37ed39e299e1-kube-api-access-42dgj\") pod \"9dfa2574-66ba-4023-b91f-37ed39e299e1\" (UID: \"9dfa2574-66ba-4023-b91f-37ed39e299e1\") "
Oct 30 13:56:18.530217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.530194    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfa2574-66ba-4023-b91f-37ed39e299e1-plugins-config-volume" pod "9dfa2574-66ba-4023-b91f-37ed39e299e1" (UID: "9dfa2574-66ba-4023-b91f-37ed39e299e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:18.530383 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.530366    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfa2574-66ba-4023-b91f-37ed39e299e1-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:18.532263 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.532237    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfa2574-66ba-4023-b91f-37ed39e299e1-kube-api-access-42dgj" pod "9dfa2574-66ba-4023-b91f-37ed39e299e1" (UID: "9dfa2574-66ba-4023-b91f-37ed39e299e1"). InnerVolumeSpecName "kube-api-access-42dgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:18.631502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.631398    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03680b2d-91b9-40e8-a3c5-5061188362b4-kube-api-access-7sck2\" (UniqueName: \"kubernetes.io/projected/03680b2d-91b9-40e8-a3c5-5061188362b4-kube-api-access-7sck2\") pod \"03680b2d-91b9-40e8-a3c5-5061188362b4\" (UID: \"03680b2d-91b9-40e8-a3c5-5061188362b4\") "
Oct 30 13:56:18.631502 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.631500    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03680b2d-91b9-40e8-a3c5-5061188362b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03680b2d-91b9-40e8-a3c5-5061188362b4-plugins-config-volume\") pod \"03680b2d-91b9-40e8-a3c5-5061188362b4\" (UID: \"03680b2d-91b9-40e8-a3c5-5061188362b4\") "
Oct 30 13:56:18.631854 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.631831    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03680b2d-91b9-40e8-a3c5-5061188362b4-plugins-config-volume" pod "03680b2d-91b9-40e8-a3c5-5061188362b4" (UID: "03680b2d-91b9-40e8-a3c5-5061188362b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:18.631938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.631919    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dgj\" (UniqueName: \"kubernetes.io/projected/9dfa2574-66ba-4023-b91f-37ed39e299e1-kube-api-access-42dgj\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:18.633724 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.633704    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03680b2d-91b9-40e8-a3c5-5061188362b4-kube-api-access-7sck2" pod "03680b2d-91b9-40e8-a3c5-5061188362b4" (UID: "03680b2d-91b9-40e8-a3c5-5061188362b4"). InnerVolumeSpecName "kube-api-access-7sck2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:18.732710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.732672    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sck2\" (UniqueName: \"kubernetes.io/projected/03680b2d-91b9-40e8-a3c5-5061188362b4-kube-api-access-7sck2\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:18.732710 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.732702    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03680b2d-91b9-40e8-a3c5-5061188362b4-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:18.806332 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.806275    4429 generic.go:356] "Generic (PLEG): container finished" podID="9dfa2574-66ba-4023-b91f-37ed39e299e1" containerID="4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33" exitCode=0
Oct 30 13:56:18.806554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.806346    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"
Oct 30 13:56:18.806554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.806384    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc" event={"ID":"9dfa2574-66ba-4023-b91f-37ed39e299e1","Type":"ContainerDied","Data":"4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33"}
Oct 30 13:56:18.806554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.806461    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc" event={"ID":"9dfa2574-66ba-4023-b91f-37ed39e299e1","Type":"ContainerDied","Data":"13f9175ecbba7484293cad7dd45a648101869255567f7842c3b4452a881f64bd"}
Oct 30 13:56:18.806554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.806494    4429 scope.go:122] "RemoveContainer" containerID="4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33"
Oct 30 13:56:18.807952 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.807928    4429 generic.go:356] "Generic (PLEG): container finished" podID="03680b2d-91b9-40e8-a3c5-5061188362b4" containerID="8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac" exitCode=0
Oct 30 13:56:18.808077 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.807971    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"
Oct 30 13:56:18.808077 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.807974    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m" event={"ID":"03680b2d-91b9-40e8-a3c5-5061188362b4","Type":"ContainerDied","Data":"8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac"}
Oct 30 13:56:18.808180 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.808088    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m" event={"ID":"03680b2d-91b9-40e8-a3c5-5061188362b4","Type":"ContainerDied","Data":"cb285cce221354575797a5351f8f99bc431ad708283c1f12d6392f9cc566a106"}
Oct 30 13:56:18.817141 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.817122    4429 scope.go:122] "RemoveContainer" containerID="4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33"
Oct 30 13:56:18.817559 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:18.817530    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33\": container with ID starting with 4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33 not found: ID does not exist" containerID="4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33"
Oct 30 13:56:18.817680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.817571    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33"} err="failed to get container status \"4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33\": rpc error: code = NotFound desc = could not find container \"4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33\": container with ID starting with 4d3a1957c560a5afdce9540d93574e4368a865cc027feab1dca8afcf0270ee33 not found: ID does not exist"
Oct 30 13:56:18.817680 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.817598    4429 scope.go:122] "RemoveContainer" containerID="8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac"
Oct 30 13:56:18.826929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.826913    4429 scope.go:122] "RemoveContainer" containerID="8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac"
Oct 30 13:56:18.827265 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:18.827240    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac\": container with ID starting with 8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac not found: ID does not exist" containerID="8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac"
Oct 30 13:56:18.827328 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.827273    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac"} err="failed to get container status \"8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac\": rpc error: code = NotFound desc = could not find container \"8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac\": container with ID starting with 8590a7355fa04ed9f5a708905e81a022256c9cd6441d1fa48128ca80ff43d6ac not found: ID does not exist"
Oct 30 13:56:18.862652 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.862618    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"]
Oct 30 13:56:18.876260 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.876224    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e652ca-7f8745bddb-kcxfc"]
Oct 30 13:56:18.913217 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.913135    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"]
Oct 30 13:56:18.919455 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:18.919423    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e652ca-68d7c9b8f9-h9q7m"]
Oct 30 13:56:20.075507 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:20.075475    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03680b2d-91b9-40e8-a3c5-5061188362b4" path="/var/lib/kubelet/pods/03680b2d-91b9-40e8-a3c5-5061188362b4/volumes"
Oct 30 13:56:20.075890 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:20.075877    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dfa2574-66ba-4023-b91f-37ed39e299e1" path="/var/lib/kubelet/pods/9dfa2574-66ba-4023-b91f-37ed39e299e1/volumes"
Oct 30 13:56:20.281153 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:20.281113    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5" podUID="e361b56b-e61d-4c0f-a915-da0415767b13" containerName="payload-pre-processing" containerID="cri-o://a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9" gracePeriod=25
Oct 30 13:56:21.565245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.565197    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"]
Oct 30 13:56:21.565834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.565505    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh" podUID="bc289b98-9450-489e-975a-b2d6ba7f5b9f" containerName="maas-api" containerID="cri-o://f4170b06343a224657bb52f90d8ad8e3e172c68bb31bd9f2933d0a7acbd04c85" gracePeriod=30
Oct 30 13:56:21.679594 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.679552    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx" podUID="bc7d3080-95f3-4fca-9680-24a2b982f3d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:21.679774 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.679563    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52" podUID="b724c857-22c4-41c1-a1f4-85548435e237" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:21.832746 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.832708    4429 generic.go:356] "Generic (PLEG): container finished" podID="bc289b98-9450-489e-975a-b2d6ba7f5b9f" containerID="f4170b06343a224657bb52f90d8ad8e3e172c68bb31bd9f2933d0a7acbd04c85" exitCode=0
Oct 30 13:56:21.833526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.832912    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh" event={"ID":"bc289b98-9450-489e-975a-b2d6ba7f5b9f","Type":"ContainerDied","Data":"f4170b06343a224657bb52f90d8ad8e3e172c68bb31bd9f2933d0a7acbd04c85"}
Oct 30 13:56:21.833526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.832999    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh" event={"ID":"bc289b98-9450-489e-975a-b2d6ba7f5b9f","Type":"ContainerDied","Data":"303580fbb4e6f23449f27dc2145ddea74efa57ff2413499f479c43e2c75c5403"}
Oct 30 13:56:21.833526 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.833023    4429 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="303580fbb4e6f23449f27dc2145ddea74efa57ff2413499f479c43e2c75c5403"
Oct 30 13:56:21.846341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.846312    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 13:56:21.967696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.967660    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc289b98-9450-489e-975a-b2d6ba7f5b9f-kube-api-access-w6hsc\" (UniqueName: \"kubernetes.io/projected/bc289b98-9450-489e-975a-b2d6ba7f5b9f-kube-api-access-w6hsc\") pod \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") "
Oct 30 13:56:21.967869 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.967735    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc289b98-9450-489e-975a-b2d6ba7f5b9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc289b98-9450-489e-975a-b2d6ba7f5b9f-openshift-service-ca\") pod \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") "
Oct 30 13:56:21.967914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.967878    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-maas-api-tls\") pod \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") "
Oct 30 13:56:21.967989 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.967961    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-metrics-certs\") pod \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\" (UID: \"bc289b98-9450-489e-975a-b2d6ba7f5b9f\") "
Oct 30 13:56:21.968185 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.968154    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc289b98-9450-489e-975a-b2d6ba7f5b9f-openshift-service-ca" pod "bc289b98-9450-489e-975a-b2d6ba7f5b9f" (UID: "bc289b98-9450-489e-975a-b2d6ba7f5b9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:21.968412 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.968389    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc289b98-9450-489e-975a-b2d6ba7f5b9f-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:21.970722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.970691    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-maas-api-tls" pod "bc289b98-9450-489e-975a-b2d6ba7f5b9f" (UID: "bc289b98-9450-489e-975a-b2d6ba7f5b9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:21.970807 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.970720    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-metrics-certs" pod "bc289b98-9450-489e-975a-b2d6ba7f5b9f" (UID: "bc289b98-9450-489e-975a-b2d6ba7f5b9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:21.970875 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:21.970855    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc289b98-9450-489e-975a-b2d6ba7f5b9f-kube-api-access-w6hsc" pod "bc289b98-9450-489e-975a-b2d6ba7f5b9f" (UID: "bc289b98-9450-489e-975a-b2d6ba7f5b9f"). InnerVolumeSpecName "kube-api-access-w6hsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:22.069063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:22.069019    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6hsc\" (UniqueName: \"kubernetes.io/projected/bc289b98-9450-489e-975a-b2d6ba7f5b9f-kube-api-access-w6hsc\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:22.069063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:22.069053    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:22.069063 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:22.069062    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc289b98-9450-489e-975a-b2d6ba7f5b9f-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:22.838814 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:22.838778    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"
Oct 30 13:56:22.897456 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:22.897418    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"]
Oct 30 13:56:22.910773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:22.910739    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e652ca-6767d8d4b8-cnjhh"]
Oct 30 13:56:24.074312 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:24.074256    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc289b98-9450-489e-975a-b2d6ba7f5b9f" path="/var/lib/kubelet/pods/bc289b98-9450-489e-975a-b2d6ba7f5b9f/volumes"
Oct 30 13:56:24.335158 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:24.335066    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5" podUID="e361b56b-e61d-4c0f-a915-da0415767b13" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:24.583179 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:24.583137    4429 scope.go:122] "RemoveContainer" containerID="f4170b06343a224657bb52f90d8ad8e3e172c68bb31bd9f2933d0a7acbd04c85"
Oct 30 13:56:25.102735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.102701    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq" podUID="cabcd6da-a086-4864-b2e7-0d41500b6c83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:25.103674 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.103649    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp" podUID="33e2e18d-e039-43aa-8822-52590f759cfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:25.213978 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.213949    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:56:25.251165 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.251123    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:56:25.304226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.304181    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b724c857-22c4-41c1-a1f4-85548435e237-kube-api-access-7k4dj\" (UniqueName: \"kubernetes.io/projected/b724c857-22c4-41c1-a1f4-85548435e237-kube-api-access-7k4dj\") pod \"b724c857-22c4-41c1-a1f4-85548435e237\" (UID: \"b724c857-22c4-41c1-a1f4-85548435e237\") "
Oct 30 13:56:25.304503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.304261    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b724c857-22c4-41c1-a1f4-85548435e237-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b724c857-22c4-41c1-a1f4-85548435e237-plugins-config-volume\") pod \"b724c857-22c4-41c1-a1f4-85548435e237\" (UID: \"b724c857-22c4-41c1-a1f4-85548435e237\") "
Oct 30 13:56:25.304984 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.304885    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b724c857-22c4-41c1-a1f4-85548435e237-plugins-config-volume" pod "b724c857-22c4-41c1-a1f4-85548435e237" (UID: "b724c857-22c4-41c1-a1f4-85548435e237"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:25.307079 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.307055    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b724c857-22c4-41c1-a1f4-85548435e237-kube-api-access-7k4dj" pod "b724c857-22c4-41c1-a1f4-85548435e237" (UID: "b724c857-22c4-41c1-a1f4-85548435e237"). InnerVolumeSpecName "kube-api-access-7k4dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:25.405741 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.405702    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7d3080-95f3-4fca-9680-24a2b982f3d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7d3080-95f3-4fca-9680-24a2b982f3d0-plugins-config-volume\") pod \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\" (UID: \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\") "
Oct 30 13:56:25.405938 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.405855    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7d3080-95f3-4fca-9680-24a2b982f3d0-kube-api-access-8zztm\" (UniqueName: \"kubernetes.io/projected/bc7d3080-95f3-4fca-9680-24a2b982f3d0-kube-api-access-8zztm\") pod \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\" (UID: \"bc7d3080-95f3-4fca-9680-24a2b982f3d0\") "
Oct 30 13:56:25.406174 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.406152    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7d3080-95f3-4fca-9680-24a2b982f3d0-plugins-config-volume" pod "bc7d3080-95f3-4fca-9680-24a2b982f3d0" (UID: "bc7d3080-95f3-4fca-9680-24a2b982f3d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:25.406338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.406311    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7d3080-95f3-4fca-9680-24a2b982f3d0-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:25.406338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.406334    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k4dj\" (UniqueName: \"kubernetes.io/projected/b724c857-22c4-41c1-a1f4-85548435e237-kube-api-access-7k4dj\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:25.406514 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.406347    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b724c857-22c4-41c1-a1f4-85548435e237-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:25.408092 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.408069    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7d3080-95f3-4fca-9680-24a2b982f3d0-kube-api-access-8zztm" pod "bc7d3080-95f3-4fca-9680-24a2b982f3d0" (UID: "bc7d3080-95f3-4fca-9680-24a2b982f3d0"). InnerVolumeSpecName "kube-api-access-8zztm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:25.507034 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.506942    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zztm\" (UniqueName: \"kubernetes.io/projected/bc7d3080-95f3-4fca-9680-24a2b982f3d0-kube-api-access-8zztm\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:25.859114 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.859079    4429 generic.go:356] "Generic (PLEG): container finished" podID="cabcd6da-a086-4864-b2e7-0d41500b6c83" containerID="a1060cb4fc582028ced69e3694438f8749e3b23a50a253b88e517c384ddce6ea" exitCode=0
Oct 30 13:56:25.859327 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.859166    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq" event={"ID":"cabcd6da-a086-4864-b2e7-0d41500b6c83","Type":"ContainerDied","Data":"a1060cb4fc582028ced69e3694438f8749e3b23a50a253b88e517c384ddce6ea"}
Oct 30 13:56:25.861129 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.861104    4429 generic.go:356] "Generic (PLEG): container finished" podID="bc7d3080-95f3-4fca-9680-24a2b982f3d0" containerID="3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6" exitCode=0
Oct 30 13:56:25.861341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.861157    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"
Oct 30 13:56:25.861341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.861157    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx" event={"ID":"bc7d3080-95f3-4fca-9680-24a2b982f3d0","Type":"ContainerDied","Data":"3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6"}
Oct 30 13:56:25.861341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.861217    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx" event={"ID":"bc7d3080-95f3-4fca-9680-24a2b982f3d0","Type":"ContainerDied","Data":"91b8ac898258bfd11130fb5cf30ae40ef5426fa8aece36cf0c318008effee404"}
Oct 30 13:56:25.861341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.861251    4429 scope.go:122] "RemoveContainer" containerID="3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6"
Oct 30 13:56:25.863474 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.863447    4429 generic.go:356] "Generic (PLEG): container finished" podID="33e2e18d-e039-43aa-8822-52590f759cfd" containerID="b328e2572e1bdcf54db55100113f21b92f0db76b63254a895aa7ed16e102f79e" exitCode=0
Oct 30 13:56:25.863641 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.863511    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp" event={"ID":"33e2e18d-e039-43aa-8822-52590f759cfd","Type":"ContainerDied","Data":"b328e2572e1bdcf54db55100113f21b92f0db76b63254a895aa7ed16e102f79e"}
Oct 30 13:56:25.865203 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.865179    4429 generic.go:356] "Generic (PLEG): container finished" podID="b724c857-22c4-41c1-a1f4-85548435e237" containerID="abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd" exitCode=0
Oct 30 13:56:25.865396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.865228    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"
Oct 30 13:56:25.865396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.865236    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52" event={"ID":"b724c857-22c4-41c1-a1f4-85548435e237","Type":"ContainerDied","Data":"abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd"}
Oct 30 13:56:25.865396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.865275    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52" event={"ID":"b724c857-22c4-41c1-a1f4-85548435e237","Type":"ContainerDied","Data":"e4aea6bdf2eb18c397573cd465c7f280dded863c6c535716db8586a43bc3017b"}
Oct 30 13:56:25.874246 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.874224    4429 scope.go:122] "RemoveContainer" containerID="3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6"
Oct 30 13:56:25.874641 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:25.874615    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6\": container with ID starting with 3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6 not found: ID does not exist" containerID="3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6"
Oct 30 13:56:25.874737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.874655    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6"} err="failed to get container status \"3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6\": rpc error: code = NotFound desc = could not find container \"3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6\": container with ID starting with 3d63d951d975736980a27b861f7731fc869f6150df2a9178b313c3e0e44162b6 not found: ID does not exist"
Oct 30 13:56:25.874737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.874684    4429 scope.go:122] "RemoveContainer" containerID="abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd"
Oct 30 13:56:25.887902 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.887879    4429 scope.go:122] "RemoveContainer" containerID="abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd"
Oct 30 13:56:25.888393 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:25.888268    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd\": container with ID starting with abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd not found: ID does not exist" containerID="abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd"
Oct 30 13:56:25.888393 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.888366    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd"} err="failed to get container status \"abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd\": rpc error: code = NotFound desc = could not find container \"abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd\": container with ID starting with abe52998538543820be689587760583d0c18c4d1d7efa44cf7d43d8add5a4dbd not found: ID does not exist"
Oct 30 13:56:25.921482 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.921417    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"]
Oct 30 13:56:25.923987 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.923955    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92855c-5cf7c45c4b-xjn52"]
Oct 30 13:56:25.964842 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.964814    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"]
Oct 30 13:56:25.967896 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.967870    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92855c-7ff68878c-grmdx"]
Oct 30 13:56:25.993337 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:25.990928    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:56:26.048885 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.048854    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:56:26.080683 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.080587    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b724c857-22c4-41c1-a1f4-85548435e237" path="/var/lib/kubelet/pods/b724c857-22c4-41c1-a1f4-85548435e237/volumes"
Oct 30 13:56:26.081015 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.080998    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7d3080-95f3-4fca-9680-24a2b982f3d0" path="/var/lib/kubelet/pods/bc7d3080-95f3-4fca-9680-24a2b982f3d0/volumes"
Oct 30 13:56:26.113684 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.113638    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabcd6da-a086-4864-b2e7-0d41500b6c83-kube-api-access-2vkf7\" (UniqueName: \"kubernetes.io/projected/cabcd6da-a086-4864-b2e7-0d41500b6c83-kube-api-access-2vkf7\") pod \"cabcd6da-a086-4864-b2e7-0d41500b6c83\" (UID: \"cabcd6da-a086-4864-b2e7-0d41500b6c83\") "
Oct 30 13:56:26.114273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.113747    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabcd6da-a086-4864-b2e7-0d41500b6c83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabcd6da-a086-4864-b2e7-0d41500b6c83-plugins-config-volume\") pod \"cabcd6da-a086-4864-b2e7-0d41500b6c83\" (UID: \"cabcd6da-a086-4864-b2e7-0d41500b6c83\") "
Oct 30 13:56:26.114273 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.114129    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabcd6da-a086-4864-b2e7-0d41500b6c83-plugins-config-volume" pod "cabcd6da-a086-4864-b2e7-0d41500b6c83" (UID: "cabcd6da-a086-4864-b2e7-0d41500b6c83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:26.115994 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.115970    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabcd6da-a086-4864-b2e7-0d41500b6c83-kube-api-access-2vkf7" pod "cabcd6da-a086-4864-b2e7-0d41500b6c83" (UID: "cabcd6da-a086-4864-b2e7-0d41500b6c83"). InnerVolumeSpecName "kube-api-access-2vkf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:26.214974 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.214938    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e2e18d-e039-43aa-8822-52590f759cfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e2e18d-e039-43aa-8822-52590f759cfd-plugins-config-volume\") pod \"33e2e18d-e039-43aa-8822-52590f759cfd\" (UID: \"33e2e18d-e039-43aa-8822-52590f759cfd\") "
Oct 30 13:56:26.215173 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.215094    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e2e18d-e039-43aa-8822-52590f759cfd-kube-api-access-5vk6b\" (UniqueName: \"kubernetes.io/projected/33e2e18d-e039-43aa-8822-52590f759cfd-kube-api-access-5vk6b\") pod \"33e2e18d-e039-43aa-8822-52590f759cfd\" (UID: \"33e2e18d-e039-43aa-8822-52590f759cfd\") "
Oct 30 13:56:26.215487 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.215454    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e2e18d-e039-43aa-8822-52590f759cfd-plugins-config-volume" pod "33e2e18d-e039-43aa-8822-52590f759cfd" (UID: "33e2e18d-e039-43aa-8822-52590f759cfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:26.215614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.215562    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33e2e18d-e039-43aa-8822-52590f759cfd-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:26.215614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.215582    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabcd6da-a086-4864-b2e7-0d41500b6c83-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:26.215614 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.215593    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vkf7\" (UniqueName: \"kubernetes.io/projected/cabcd6da-a086-4864-b2e7-0d41500b6c83-kube-api-access-2vkf7\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:26.217863 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.217838    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e2e18d-e039-43aa-8822-52590f759cfd-kube-api-access-5vk6b" pod "33e2e18d-e039-43aa-8822-52590f759cfd" (UID: "33e2e18d-e039-43aa-8822-52590f759cfd"). InnerVolumeSpecName "kube-api-access-5vk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:26.230108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.230074    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"]
Oct 30 13:56:26.316881 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.316834    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vk6b\" (UniqueName: \"kubernetes.io/projected/33e2e18d-e039-43aa-8822-52590f759cfd-kube-api-access-5vk6b\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:26.872944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.872914    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"
Oct 30 13:56:26.872944 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.872920    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp" event={"ID":"33e2e18d-e039-43aa-8822-52590f759cfd","Type":"ContainerDied","Data":"eb8b8028d71725673e942759f2c890272702efbc572da15958faedb5ca440e66"}
Oct 30 13:56:26.873213 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.873012    4429 scope.go:122] "RemoveContainer" containerID="b328e2572e1bdcf54db55100113f21b92f0db76b63254a895aa7ed16e102f79e"
Oct 30 13:56:26.875717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.875686    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq" event={"ID":"cabcd6da-a086-4864-b2e7-0d41500b6c83","Type":"ContainerDied","Data":"99406c0c6e8798790b73b4f83693380d731259fadb163433cece15bf652096ea"}
Oct 30 13:56:26.875850 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.875798    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"
Oct 30 13:56:26.883529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.883507    4429 scope.go:122] "RemoveContainer" containerID="a1060cb4fc582028ced69e3694438f8749e3b23a50a253b88e517c384ddce6ea"
Oct 30 13:56:26.930470 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.930418    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"]
Oct 30 13:56:26.935186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.935157    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-517a25-fcdcc76d7-nmcmp"]
Oct 30 13:56:26.977372 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.977338    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"]
Oct 30 13:56:26.979403 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:26.979378    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-517a25-67d96956bb-p7jgq"]
Oct 30 13:56:28.075554 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:28.075522    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e2e18d-e039-43aa-8822-52590f759cfd" path="/var/lib/kubelet/pods/33e2e18d-e039-43aa-8822-52590f759cfd/volumes"
Oct 30 13:56:28.076042 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:28.076026    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabcd6da-a086-4864-b2e7-0d41500b6c83" path="/var/lib/kubelet/pods/cabcd6da-a086-4864-b2e7-0d41500b6c83/volumes"
Oct 30 13:56:30.110674 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:30.110641    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"]
Oct 30 13:56:31.231164 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:31.231113    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq" podUID="94e10d3e-d608-45bb-bfc5-f8905b852890" containerName="payload-pre-processing" containerID="cri-o://3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b" gracePeriod=25
Oct 30 13:56:34.335478 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:34.335440    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5" podUID="e361b56b-e61d-4c0f-a915-da0415767b13" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:35.111965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:35.111925    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6" podUID="9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" containerName="payload-pre-processing" containerID="cri-o://a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c" gracePeriod=25
Oct 30 13:56:38.528143 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.528107    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"]
Oct 30 13:56:38.528705 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.528348    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx" podUID="3c5b7fde-8638-4360-b709-58c7f03ca5e2" containerName="maas-api" containerID="cri-o://b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6" gracePeriod=30
Oct 30 13:56:38.799981 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.799890    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 13:56:38.857579 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.857545    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-metrics-certs\") pod \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") "
Oct 30 13:56:38.857765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.857604    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-maas-api-tls\") pod \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") "
Oct 30 13:56:38.857765 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.857659    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5b7fde-8638-4360-b709-58c7f03ca5e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b7fde-8638-4360-b709-58c7f03ca5e2-openshift-service-ca\") pod \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") "
Oct 30 13:56:38.857857 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.857779    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5b7fde-8638-4360-b709-58c7f03ca5e2-kube-api-access-m9g7v\" (UniqueName: \"kubernetes.io/projected/3c5b7fde-8638-4360-b709-58c7f03ca5e2-kube-api-access-m9g7v\") pod \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\" (UID: \"3c5b7fde-8638-4360-b709-58c7f03ca5e2\") "
Oct 30 13:56:38.858172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.858140    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5b7fde-8638-4360-b709-58c7f03ca5e2-openshift-service-ca" pod "3c5b7fde-8638-4360-b709-58c7f03ca5e2" (UID: "3c5b7fde-8638-4360-b709-58c7f03ca5e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:38.858331 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.858278    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b7fde-8638-4360-b709-58c7f03ca5e2-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:38.860087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.860046    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-maas-api-tls" pod "3c5b7fde-8638-4360-b709-58c7f03ca5e2" (UID: "3c5b7fde-8638-4360-b709-58c7f03ca5e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:38.860229 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.860102    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-metrics-certs" pod "3c5b7fde-8638-4360-b709-58c7f03ca5e2" (UID: "3c5b7fde-8638-4360-b709-58c7f03ca5e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:38.860505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.860488    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5b7fde-8638-4360-b709-58c7f03ca5e2-kube-api-access-m9g7v" pod "3c5b7fde-8638-4360-b709-58c7f03ca5e2" (UID: "3c5b7fde-8638-4360-b709-58c7f03ca5e2"). InnerVolumeSpecName "kube-api-access-m9g7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:38.958961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.958932    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:38.958961 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.958960    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5b7fde-8638-4360-b709-58c7f03ca5e2-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:38.959216 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.958972    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9g7v\" (UniqueName: \"kubernetes.io/projected/3c5b7fde-8638-4360-b709-58c7f03ca5e2-kube-api-access-m9g7v\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:38.959530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.959505    4429 generic.go:356] "Generic (PLEG): container finished" podID="3c5b7fde-8638-4360-b709-58c7f03ca5e2" containerID="b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6" exitCode=0
Oct 30 13:56:38.959630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.959558    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"
Oct 30 13:56:38.959630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.959585    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx" event={"ID":"3c5b7fde-8638-4360-b709-58c7f03ca5e2","Type":"ContainerDied","Data":"b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6"}
Oct 30 13:56:38.959740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.959652    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx" event={"ID":"3c5b7fde-8638-4360-b709-58c7f03ca5e2","Type":"ContainerDied","Data":"22e82cc4dad37bbb5997ab6d6a346ab05f1b5faeffef9a0cc391ff15827362ec"}
Oct 30 13:56:38.959740 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.959684    4429 scope.go:122] "RemoveContainer" containerID="b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6"
Oct 30 13:56:38.970045 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.970020    4429 scope.go:122] "RemoveContainer" containerID="b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6"
Oct 30 13:56:38.970401 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:38.970372    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6\": container with ID starting with b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6 not found: ID does not exist" containerID="b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6"
Oct 30 13:56:38.970491 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:38.970411    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6"} err="failed to get container status \"b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6\": rpc error: code = NotFound desc = could not find container \"b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6\": container with ID starting with b87fa75ae8750bc695aa7e65ba0f5ea993cd76dbe3cd50e4d67369e91a0feaf6 not found: ID does not exist"
Oct 30 13:56:39.019719 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:39.019688    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"]
Oct 30 13:56:39.027501 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:39.027471    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-bb86fd-7cb8dfd886-x26dx"]
Oct 30 13:56:39.627358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:39.627320    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq" podUID="94e10d3e-d608-45bb-bfc5-f8905b852890" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:40.074664 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.074623    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5b7fde-8638-4360-b709-58c7f03ca5e2" path="/var/lib/kubelet/pods/3c5b7fde-8638-4360-b709-58c7f03ca5e2/volumes"
Oct 30 13:56:40.536539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.536514    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:56:40.676549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.676441    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e361b56b-e61d-4c0f-a915-da0415767b13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e361b56b-e61d-4c0f-a915-da0415767b13-plugins-config-volume\") pod \"e361b56b-e61d-4c0f-a915-da0415767b13\" (UID: \"e361b56b-e61d-4c0f-a915-da0415767b13\") "
Oct 30 13:56:40.677022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.676577    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e361b56b-e61d-4c0f-a915-da0415767b13-kube-api-access-j92mz\" (UniqueName: \"kubernetes.io/projected/e361b56b-e61d-4c0f-a915-da0415767b13-kube-api-access-j92mz\") pod \"e361b56b-e61d-4c0f-a915-da0415767b13\" (UID: \"e361b56b-e61d-4c0f-a915-da0415767b13\") "
Oct 30 13:56:40.677022 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.676863    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e361b56b-e61d-4c0f-a915-da0415767b13-plugins-config-volume" pod "e361b56b-e61d-4c0f-a915-da0415767b13" (UID: "e361b56b-e61d-4c0f-a915-da0415767b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:40.678830 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.678806    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e361b56b-e61d-4c0f-a915-da0415767b13-kube-api-access-j92mz" pod "e361b56b-e61d-4c0f-a915-da0415767b13" (UID: "e361b56b-e61d-4c0f-a915-da0415767b13"). InnerVolumeSpecName "kube-api-access-j92mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:40.778465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.778429    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92mz\" (UniqueName: \"kubernetes.io/projected/e361b56b-e61d-4c0f-a915-da0415767b13-kube-api-access-j92mz\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:40.778465 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.778468    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e361b56b-e61d-4c0f-a915-da0415767b13-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:40.973175 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.973142    4429 generic.go:356] "Generic (PLEG): container finished" podID="e361b56b-e61d-4c0f-a915-da0415767b13" containerID="a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9" exitCode=0
Oct 30 13:56:40.973443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.973201    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"
Oct 30 13:56:40.973443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.973236    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5" event={"ID":"e361b56b-e61d-4c0f-a915-da0415767b13","Type":"ContainerDied","Data":"a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9"}
Oct 30 13:56:40.973443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.973315    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5" event={"ID":"e361b56b-e61d-4c0f-a915-da0415767b13","Type":"ContainerDied","Data":"cb17be68249630ef95b4495849af32067a298c188453f2cd01b6459448a2297d"}
Oct 30 13:56:40.973443 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.973347    4429 scope.go:122] "RemoveContainer" containerID="a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9"
Oct 30 13:56:40.983120 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.983103    4429 scope.go:122] "RemoveContainer" containerID="a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9"
Oct 30 13:56:40.983482 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:40.983462    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9\": container with ID starting with a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9 not found: ID does not exist" containerID="a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9"
Oct 30 13:56:40.983573 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:40.983489    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9"} err="failed to get container status \"a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9\": rpc error: code = NotFound desc = could not find container \"a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9\": container with ID starting with a4e972d5b1a199d8f8393f1b0c427ff243c6f967209bbd8f49a33a053d3f33c9 not found: ID does not exist"
Oct 30 13:56:41.030381 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:41.030346    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"]
Oct 30 13:56:41.034043 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:41.034010    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-75d385-67669b7955-t2gd5"]
Oct 30 13:56:42.074473 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:42.074440    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e361b56b-e61d-4c0f-a915-da0415767b13" path="/var/lib/kubelet/pods/e361b56b-e61d-4c0f-a915-da0415767b13/volumes"
Oct 30 13:56:42.372189 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:42.372104    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:56:44.903205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:44.903166    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6" podUID="9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:47.587025 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:47.586983    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"]
Oct 30 13:56:47.620717 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:47.620672    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"]
Oct 30 13:56:49.626587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:49.626537    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq" podUID="94e10d3e-d608-45bb-bfc5-f8905b852890" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:49.982370 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:49.982326    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"]
Oct 30 13:56:49.982587 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:49.982562    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675" podUID="4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" containerName="maas-api" containerID="cri-o://b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d" gracePeriod=30
Oct 30 13:56:50.255907 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.255829    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:56:50.373951 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.373915    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls\") pod \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") "
Oct 30 13:56:50.374193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.373995    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-openshift-service-ca\") pod \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") "
Oct 30 13:56:50.374193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.374026    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-metrics-certs\") pod \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") "
Oct 30 13:56:50.374193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.374110    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-kube-api-access-ntgj8\" (UniqueName: \"kubernetes.io/projected/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-kube-api-access-ntgj8\") pod \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\" (UID: \"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6\") "
Oct 30 13:56:50.374468 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.374439    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-openshift-service-ca" pod "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" (UID: "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:50.374611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.374566    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:50.376419 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.376392    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls" pod "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" (UID: "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:50.376530 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.376412    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-metrics-certs" pod "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" (UID: "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:56:50.376595 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.376541    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-kube-api-access-ntgj8" pod "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" (UID: "4f1282ee-48bd-49fa-8af5-5c22dc7abfc6"). InnerVolumeSpecName "kube-api-access-ntgj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:50.475648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.475610    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntgj8\" (UniqueName: \"kubernetes.io/projected/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-kube-api-access-ntgj8\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:50.475648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.475645    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:50.475648 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:50.475654    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:51.030193 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.030160    4429 generic.go:356] "Generic (PLEG): container finished" podID="4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" containerID="b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d" exitCode=0
Oct 30 13:56:51.030679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.030214    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"
Oct 30 13:56:51.030679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.030243    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675" event={"ID":"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6","Type":"ContainerDied","Data":"b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d"}
Oct 30 13:56:51.030679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.030326    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675" event={"ID":"4f1282ee-48bd-49fa-8af5-5c22dc7abfc6","Type":"ContainerDied","Data":"5383a5a27bd20f66a2443c96eeaf8f9f1734013e86cc810e7f5be71a64ad7414"}
Oct 30 13:56:51.030679 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.030351    4429 scope.go:122] "RemoveContainer" containerID="b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d"
Oct 30 13:56:51.040796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.040767    4429 scope.go:122] "RemoveContainer" containerID="b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d"
Oct 30 13:56:51.041143 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:51.041123    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d\": container with ID starting with b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d not found: ID does not exist" containerID="b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d"
Oct 30 13:56:51.041223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.041153    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d"} err="failed to get container status \"b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d\": rpc error: code = NotFound desc = could not find container \"b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d\": container with ID starting with b677cfa7937e9279116fd319cb56ccf41d8403fa27aa552257b7232bcd28537d not found: ID does not exist"
Oct 30 13:56:51.089214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.089171    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"]
Oct 30 13:56:51.102602 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.102566    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f49b40-57d7cc65bc-hx675"]
Oct 30 13:56:51.500036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.500006    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 13:56:51.586834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.586795    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e10d3e-d608-45bb-bfc5-f8905b852890-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e10d3e-d608-45bb-bfc5-f8905b852890-plugins-config-volume\") pod \"94e10d3e-d608-45bb-bfc5-f8905b852890\" (UID: \"94e10d3e-d608-45bb-bfc5-f8905b852890\") "
Oct 30 13:56:51.586834 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.586841    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e10d3e-d608-45bb-bfc5-f8905b852890-kube-api-access-xl9dz\" (UniqueName: \"kubernetes.io/projected/94e10d3e-d608-45bb-bfc5-f8905b852890-kube-api-access-xl9dz\") pod \"94e10d3e-d608-45bb-bfc5-f8905b852890\" (UID: \"94e10d3e-d608-45bb-bfc5-f8905b852890\") "
Oct 30 13:56:51.587321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.587264    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e10d3e-d608-45bb-bfc5-f8905b852890-plugins-config-volume" pod "94e10d3e-d608-45bb-bfc5-f8905b852890" (UID: "94e10d3e-d608-45bb-bfc5-f8905b852890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:51.589368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.589344    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e10d3e-d608-45bb-bfc5-f8905b852890-kube-api-access-xl9dz" pod "94e10d3e-d608-45bb-bfc5-f8905b852890" (UID: "94e10d3e-d608-45bb-bfc5-f8905b852890"). InnerVolumeSpecName "kube-api-access-xl9dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:51.688226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.688185    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e10d3e-d608-45bb-bfc5-f8905b852890-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:51.688226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:51.688222    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl9dz\" (UniqueName: \"kubernetes.io/projected/94e10d3e-d608-45bb-bfc5-f8905b852890-kube-api-access-xl9dz\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:52.037407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.037308    4429 generic.go:356] "Generic (PLEG): container finished" podID="94e10d3e-d608-45bb-bfc5-f8905b852890" containerID="3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b" exitCode=0
Oct 30 13:56:52.037407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.037372    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"
Oct 30 13:56:52.037407 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.037396    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq" event={"ID":"94e10d3e-d608-45bb-bfc5-f8905b852890","Type":"ContainerDied","Data":"3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b"}
Oct 30 13:56:52.038055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.037456    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq" event={"ID":"94e10d3e-d608-45bb-bfc5-f8905b852890","Type":"ContainerDied","Data":"83517657f9e84e8a06766713cbc5c1597262ae42a00f7fd33d2d259299e51e05"}
Oct 30 13:56:52.038055 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.037483    4429 scope.go:122] "RemoveContainer" containerID="3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b"
Oct 30 13:56:52.047609 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.047574    4429 scope.go:122] "RemoveContainer" containerID="3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b"
Oct 30 13:56:52.047950 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:52.047930    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b\": container with ID starting with 3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b not found: ID does not exist" containerID="3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b"
Oct 30 13:56:52.047995 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.047962    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b"} err="failed to get container status \"3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b\": rpc error: code = NotFound desc = could not find container \"3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b\": container with ID starting with 3568dba4aef65ee5f8cfde71536ddc3e67587c7c32a5e671f51a4741fb5f4e3b not found: ID does not exist"
Oct 30 13:56:52.076157 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.076122    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" path="/var/lib/kubelet/pods/4f1282ee-48bd-49fa-8af5-5c22dc7abfc6/volumes"
Oct 30 13:56:52.092604 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.092570    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"]
Oct 30 13:56:52.098619 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.098583    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-bb86fd-6b8dcfc5f9-4kzgq"]
Oct 30 13:56:52.587833 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.587785    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x" podUID="b7562f85-16f9-466c-8814-42b7c8307db1" containerName="payload-processing" containerID="cri-o://187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0" gracePeriod=25
Oct 30 13:56:52.621682 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:52.621629    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw" podUID="bc86c779-342a-4ad1-bbe5-8a3cda3983a3" containerName="payload-pre-processing" containerID="cri-o://358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93" gracePeriod=25
Oct 30 13:56:54.094662 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:54.094626    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e10d3e-d608-45bb-bfc5-f8905b852890" path="/var/lib/kubelet/pods/94e10d3e-d608-45bb-bfc5-f8905b852890/volumes"
Oct 30 13:56:54.902516 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:54.902476    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6" podUID="9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:55.372535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:55.372508    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:56:55.550450 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:55.550376    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-kube-api-access-8g5zg\" (UniqueName: \"kubernetes.io/projected/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-kube-api-access-8g5zg\") pod \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\" (UID: \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\") "
Oct 30 13:56:55.550617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:55.550474    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-plugins-config-volume\") pod \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\" (UID: \"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d\") "
Oct 30 13:56:55.550908 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:55.550878    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-plugins-config-volume" pod "9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" (UID: "9a5b7987-fc95-4e40-9acf-6f4b52b68a8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:56:55.552735 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:55.552703    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-kube-api-access-8g5zg" pod "9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" (UID: "9a5b7987-fc95-4e40-9acf-6f4b52b68a8d"). InnerVolumeSpecName "kube-api-access-8g5zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:56:55.652069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:55.652039    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g5zg\" (UniqueName: \"kubernetes.io/projected/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-kube-api-access-8g5zg\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:55.652069 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:55.652065    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:56:56.063598 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.063558    4429 generic.go:356] "Generic (PLEG): container finished" podID="9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" containerID="a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c" exitCode=0
Oct 30 13:56:56.063808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.063604    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6" event={"ID":"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d","Type":"ContainerDied","Data":"a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c"}
Oct 30 13:56:56.063808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.063616    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"
Oct 30 13:56:56.063808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.063655    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6" event={"ID":"9a5b7987-fc95-4e40-9acf-6f4b52b68a8d","Type":"ContainerDied","Data":"0f076d0567e90ccc4704474216d0236aeade0e3c2eeb00ffbe9b4c630c4d542e"}
Oct 30 13:56:56.063808 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.063677    4429 scope.go:122] "RemoveContainer" containerID="a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c"
Oct 30 13:56:56.074186 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.074168    4429 scope.go:122] "RemoveContainer" containerID="a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c"
Oct 30 13:56:56.074456 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:56:56.074437    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c\": container with ID starting with a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c not found: ID does not exist" containerID="a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c"
Oct 30 13:56:56.074545 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.074463    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c"} err="failed to get container status \"a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c\": rpc error: code = NotFound desc = could not find container \"a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c\": container with ID starting with a83e920a2ffaf3475a479e76e683820fe016c42e54b2cb90edc19a0eb34bac7c not found: ID does not exist"
Oct 30 13:56:56.120102 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.120077    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"]
Oct 30 13:56:56.133674 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.133654    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-241ffa-78b5645f79-hlbp6"]
Oct 30 13:56:56.474750 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.474712    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw" podUID="bc86c779-342a-4ad1-bbe5-8a3cda3983a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:56.475149 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:56.474818    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x" podUID="b7562f85-16f9-466c-8814-42b7c8307db1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:56:58.074448 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:58.074414    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" path="/var/lib/kubelet/pods/9a5b7987-fc95-4e40-9acf-6f4b52b68a8d/volumes"
Oct 30 13:56:59.410082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:59.410047    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"]
Oct 30 13:56:59.410592 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:56:59.410326    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy" containerID="cri-o://ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950" gracePeriod=30
Oct 30 13:57:00.443357 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:00.443322    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"]
Oct 30 13:57:04.670294 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.670257    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:57:04.735401 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735370    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-istio-token\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-istio-token\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.735629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735427    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-data\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.735629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735475    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-certs\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.735629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735511    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-envoy\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.735629 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735557    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f6dd987-feca-427c-8f9b-724c899c2678-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6dd987-feca-427c-8f9b-724c899c2678-istio-podinfo\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.735822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735635    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6dd987-feca-427c-8f9b-724c899c2678-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6dd987-feca-427c-8f9b-724c899c2678-istiod-ca-cert\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.735822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735676    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-credential-socket\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.735822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735721    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-socket\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.735822 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.735764    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-kube-api-access-ghgkq\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-kube-api-access-ghgkq\") pod \"0f6dd987-feca-427c-8f9b-724c899c2678\" (UID: \"0f6dd987-feca-427c-8f9b-724c899c2678\") "
Oct 30 13:57:04.736720 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.736360    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-data" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:04.737132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.736893    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6dd987-feca-427c-8f9b-724c899c2678-istiod-ca-cert" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:04.737132 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.736948    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-credential-socket" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:04.737363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.737156    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-certs" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:04.737363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.737183    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-socket" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:04.738818 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.738782    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f6dd987-feca-427c-8f9b-724c899c2678-istio-podinfo" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:57:04.738929 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.738781    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-istio-token" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:04.739364 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.739339    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-kube-api-access-ghgkq" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "kube-api-access-ghgkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:04.739889 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.739869    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-envoy" pod "0f6dd987-feca-427c-8f9b-724c899c2678" (UID: "0f6dd987-feca-427c-8f9b-724c899c2678"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:04.837075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837041    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.837075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837071    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghgkq\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-kube-api-access-ghgkq\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.837075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837083    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6dd987-feca-427c-8f9b-724c899c2678-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.837358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837092    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.837358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837100    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.837358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837107    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.837358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837115    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6dd987-feca-427c-8f9b-724c899c2678-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.837358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837122    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6dd987-feca-427c-8f9b-724c899c2678-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:04.837358 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:04.837132    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6dd987-feca-427c-8f9b-724c899c2678-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:05.112569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.112470    4429 generic.go:356] "Generic (PLEG): container finished" podID="0f6dd987-feca-427c-8f9b-724c899c2678" containerID="ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950" exitCode=0
Oct 30 13:57:05.112569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.112539    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"
Oct 30 13:57:05.112569 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.112556    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" event={"ID":"0f6dd987-feca-427c-8f9b-724c899c2678","Type":"ContainerDied","Data":"ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950"}
Oct 30 13:57:05.112882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.112615    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth" event={"ID":"0f6dd987-feca-427c-8f9b-724c899c2678","Type":"ContainerDied","Data":"19e2e95c45fc43619d88e4d02147d412bb1ccd4bac496bb15567fe31d8be2b77"}
Oct 30 13:57:05.112882 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.112638    4429 scope.go:122] "RemoveContainer" containerID="ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950"
Oct 30 13:57:05.122362 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.122347    4429 scope.go:122] "RemoveContainer" containerID="7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb"
Oct 30 13:57:05.131520 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.131495    4429 scope.go:122] "RemoveContainer" containerID="ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950"
Oct 30 13:57:05.131882 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:05.131859    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950\": container with ID starting with ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950 not found: ID does not exist" containerID="ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950"
Oct 30 13:57:05.131970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.131892    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950"} err="failed to get container status \"ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950\": rpc error: code = NotFound desc = could not find container \"ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950\": container with ID starting with ad7a0828679dc95938bc46784087068637636c1449412cc04f4f124783dc5950 not found: ID does not exist"
Oct 30 13:57:05.131970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.131911    4429 scope.go:122] "RemoveContainer" containerID="7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb"
Oct 30 13:57:05.132160 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:05.132141    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb\": container with ID starting with 7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb not found: ID does not exist" containerID="7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb"
Oct 30 13:57:05.132199 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.132171    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb"} err="failed to get container status \"7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb\": rpc error: code = NotFound desc = could not find container \"7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb\": container with ID starting with 7e570bcc7e7a281bfbbf964f92d793b6c709131c9d9682298e8fbb6779da51bb not found: ID does not exist"
Oct 30 13:57:05.169243 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.169212    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"]
Oct 30 13:57:05.183139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.183112    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f49b40-openshift-default-54759894c5-n6bth"]
Oct 30 13:57:05.443713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.443677    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft" podUID="b1b1df1d-e2a2-4fe2-8598-49573af73613" containerName="payload-pre-processing" containerID="cri-o://eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66" gracePeriod=25
Oct 30 13:57:05.696768 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.696709    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 13:57:05.746849 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.746805    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b1df1d-e2a2-4fe2-8598-49573af73613-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b1df1d-e2a2-4fe2-8598-49573af73613-plugins-config-volume\") pod \"b1b1df1d-e2a2-4fe2-8598-49573af73613\" (UID: \"b1b1df1d-e2a2-4fe2-8598-49573af73613\") "
Oct 30 13:57:05.747326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.746964    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b1df1d-e2a2-4fe2-8598-49573af73613-kube-api-access-z5prs\" (UniqueName: \"kubernetes.io/projected/b1b1df1d-e2a2-4fe2-8598-49573af73613-kube-api-access-z5prs\") pod \"b1b1df1d-e2a2-4fe2-8598-49573af73613\" (UID: \"b1b1df1d-e2a2-4fe2-8598-49573af73613\") "
Oct 30 13:57:05.747326 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.747232    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b1df1d-e2a2-4fe2-8598-49573af73613-plugins-config-volume" pod "b1b1df1d-e2a2-4fe2-8598-49573af73613" (UID: "b1b1df1d-e2a2-4fe2-8598-49573af73613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:05.749325 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.749268    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b1df1d-e2a2-4fe2-8598-49573af73613-kube-api-access-z5prs" pod "b1b1df1d-e2a2-4fe2-8598-49573af73613" (UID: "b1b1df1d-e2a2-4fe2-8598-49573af73613"). InnerVolumeSpecName "kube-api-access-z5prs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:05.848163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.848125    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5prs\" (UniqueName: \"kubernetes.io/projected/b1b1df1d-e2a2-4fe2-8598-49573af73613-kube-api-access-z5prs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:05.848163 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:05.848156    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b1df1d-e2a2-4fe2-8598-49573af73613-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:06.073764 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.073673    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" path="/var/lib/kubelet/pods/0f6dd987-feca-427c-8f9b-724c899c2678/volumes"
Oct 30 13:57:06.118525 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.118493    4429 generic.go:356] "Generic (PLEG): container finished" podID="b1b1df1d-e2a2-4fe2-8598-49573af73613" containerID="eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66" exitCode=0
Oct 30 13:57:06.118738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.118533    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"
Oct 30 13:57:06.118738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.118563    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft" event={"ID":"b1b1df1d-e2a2-4fe2-8598-49573af73613","Type":"ContainerDied","Data":"eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66"}
Oct 30 13:57:06.118738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.118616    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft" event={"ID":"b1b1df1d-e2a2-4fe2-8598-49573af73613","Type":"ContainerDied","Data":"1812a155c0da2189468727751168b1efd719b3004ad2c419753efe2475e312a1"}
Oct 30 13:57:06.118738 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.118639    4429 scope.go:122] "RemoveContainer" containerID="eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66"
Oct 30 13:57:06.127550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.127531    4429 scope.go:122] "RemoveContainer" containerID="eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66"
Oct 30 13:57:06.127859 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:06.127830    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66\": container with ID starting with eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66 not found: ID does not exist" containerID="eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66"
Oct 30 13:57:06.127965 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.127859    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66"} err="failed to get container status \"eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66\": rpc error: code = NotFound desc = could not find container \"eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66\": container with ID starting with eb8cf3020f2252cbffa7c657e031610d8081210f533ddcc83cea0698b1b73e66 not found: ID does not exist"
Oct 30 13:57:06.175035 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.175002    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"]
Oct 30 13:57:06.180865 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.180837    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7a8b03-67b9cbb595-6krft"]
Oct 30 13:57:06.475356 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.475323    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw" podUID="bc86c779-342a-4ad1-bbe5-8a3cda3983a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:57:06.475541 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:06.475364    4429 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x" podUID="b7562f85-16f9-466c-8814-42b7c8307db1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:57:08.082266 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:08.082226    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b1df1d-e2a2-4fe2-8598-49573af73613" path="/var/lib/kubelet/pods/b1b1df1d-e2a2-4fe2-8598-49573af73613/volumes"
Oct 30 13:57:12.370894 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:12.370860    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:57:12.874653 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.874630    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:57:12.899082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.899056    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:57:12.921172 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.921093    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7562f85-16f9-466c-8814-42b7c8307db1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7562f85-16f9-466c-8814-42b7c8307db1-plugins-config-volume\") pod \"b7562f85-16f9-466c-8814-42b7c8307db1\" (UID: \"b7562f85-16f9-466c-8814-42b7c8307db1\") "
Oct 30 13:57:12.921345 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.921267    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7562f85-16f9-466c-8814-42b7c8307db1-kube-api-access-lf2cs\" (UniqueName: \"kubernetes.io/projected/b7562f85-16f9-466c-8814-42b7c8307db1-kube-api-access-lf2cs\") pod \"b7562f85-16f9-466c-8814-42b7c8307db1\" (UID: \"b7562f85-16f9-466c-8814-42b7c8307db1\") "
Oct 30 13:57:12.921558 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.921533    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7562f85-16f9-466c-8814-42b7c8307db1-plugins-config-volume" pod "b7562f85-16f9-466c-8814-42b7c8307db1" (UID: "b7562f85-16f9-466c-8814-42b7c8307db1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:12.921672 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.921652    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7562f85-16f9-466c-8814-42b7c8307db1-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:12.923935 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.923910    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7562f85-16f9-466c-8814-42b7c8307db1-kube-api-access-lf2cs" pod "b7562f85-16f9-466c-8814-42b7c8307db1" (UID: "b7562f85-16f9-466c-8814-42b7c8307db1"). InnerVolumeSpecName "kube-api-access-lf2cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:12.988453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.988154    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"]
Oct 30 13:57:12.988453 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:12.988434    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m" podUID="3c196508-b0b5-44eb-8be9-190309a6184e" containerName="maas-api" containerID="cri-o://dab3f4768e231ef324d3e4e8adbf5870d94ddb17706f0a80421cd868ccc5cf12" gracePeriod=30
Oct 30 13:57:13.022170 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.022146    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-plugins-config-volume\") pod \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\" (UID: \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\") "
Oct 30 13:57:13.022336 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.022247    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-kube-api-access-5pj8l\" (UniqueName: \"kubernetes.io/projected/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-kube-api-access-5pj8l\") pod \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\" (UID: \"bc86c779-342a-4ad1-bbe5-8a3cda3983a3\") "
Oct 30 13:57:13.022528 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.022507    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-plugins-config-volume" pod "bc86c779-342a-4ad1-bbe5-8a3cda3983a3" (UID: "bc86c779-342a-4ad1-bbe5-8a3cda3983a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:13.022665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.022646    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:13.022665 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.022663    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2cs\" (UniqueName: \"kubernetes.io/projected/b7562f85-16f9-466c-8814-42b7c8307db1-kube-api-access-lf2cs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:13.024391 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.024361    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-kube-api-access-5pj8l" pod "bc86c779-342a-4ad1-bbe5-8a3cda3983a3" (UID: "bc86c779-342a-4ad1-bbe5-8a3cda3983a3"). InnerVolumeSpecName "kube-api-access-5pj8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:13.123321 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.123276    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pj8l\" (UniqueName: \"kubernetes.io/projected/bc86c779-342a-4ad1-bbe5-8a3cda3983a3-kube-api-access-5pj8l\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:13.159067 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.159032    4429 generic.go:356] "Generic (PLEG): container finished" podID="b7562f85-16f9-466c-8814-42b7c8307db1" containerID="187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0" exitCode=0
Oct 30 13:57:13.159228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.159097    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x" event={"ID":"b7562f85-16f9-466c-8814-42b7c8307db1","Type":"ContainerDied","Data":"187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0"}
Oct 30 13:57:13.159228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.159150    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"
Oct 30 13:57:13.159228 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.159166    4429 scope.go:122] "RemoveContainer" containerID="187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0"
Oct 30 13:57:13.159426 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.159155    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x" event={"ID":"b7562f85-16f9-466c-8814-42b7c8307db1","Type":"ContainerDied","Data":"74ae2d5b2be07cdca1ebe5e67a628cca4cc23cfcb9af6fce9c2cc9ee1ef67248"}
Oct 30 13:57:13.161363 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.161209    4429 generic.go:356] "Generic (PLEG): container finished" podID="bc86c779-342a-4ad1-bbe5-8a3cda3983a3" containerID="358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93" exitCode=0
Oct 30 13:57:13.161492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.161362    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw" event={"ID":"bc86c779-342a-4ad1-bbe5-8a3cda3983a3","Type":"ContainerDied","Data":"358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93"}
Oct 30 13:57:13.161492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.161407    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw" event={"ID":"bc86c779-342a-4ad1-bbe5-8a3cda3983a3","Type":"ContainerDied","Data":"c1e82f3577e801d3d6cccca3cc875a8c492e3a9a6068e50986bfb703d4255310"}
Oct 30 13:57:13.161492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.161486    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"
Oct 30 13:57:13.164986 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.164962    4429 generic.go:356] "Generic (PLEG): container finished" podID="3c196508-b0b5-44eb-8be9-190309a6184e" containerID="dab3f4768e231ef324d3e4e8adbf5870d94ddb17706f0a80421cd868ccc5cf12" exitCode=0
Oct 30 13:57:13.165094 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.165043    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m" event={"ID":"3c196508-b0b5-44eb-8be9-190309a6184e","Type":"ContainerDied","Data":"dab3f4768e231ef324d3e4e8adbf5870d94ddb17706f0a80421cd868ccc5cf12"}
Oct 30 13:57:13.174899 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.174877    4429 scope.go:122] "RemoveContainer" containerID="187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0"
Oct 30 13:57:13.175236 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:13.175214    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0\": container with ID starting with 187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0 not found: ID does not exist" containerID="187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0"
Oct 30 13:57:13.175338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.175245    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0"} err="failed to get container status \"187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0\": rpc error: code = NotFound desc = could not find container \"187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0\": container with ID starting with 187d36b99f4ef072f68ec81eb41169969be0151d000aa6a250a731f71fabfeb0 not found: ID does not exist"
Oct 30 13:57:13.175338 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.175261    4429 scope.go:122] "RemoveContainer" containerID="358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93"
Oct 30 13:57:13.184855 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.184832    4429 scope.go:122] "RemoveContainer" containerID="358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93"
Oct 30 13:57:13.185183 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:13.185162    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93\": container with ID starting with 358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93 not found: ID does not exist" containerID="358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93"
Oct 30 13:57:13.185278 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.185190    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93"} err="failed to get container status \"358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93\": rpc error: code = NotFound desc = could not find container \"358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93\": container with ID starting with 358f7918fd23364f3b620dfe0d040162264ecb0ecc1432659cfcc6b1d2648a93 not found: ID does not exist"
Oct 30 13:57:13.222333 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.222300    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"]
Oct 30 13:57:13.225584 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.225562    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f49b40-7bcf54d65b-vqj2x"]
Oct 30 13:57:13.240486 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.240463    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:57:13.264928 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.264903    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"]
Oct 30 13:57:13.268434 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.268413    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f49b40-565d6f5f87-xppfw"]
Oct 30 13:57:13.325121 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.325094    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c196508-b0b5-44eb-8be9-190309a6184e-kube-api-access-djdst\" (UniqueName: \"kubernetes.io/projected/3c196508-b0b5-44eb-8be9-190309a6184e-kube-api-access-djdst\") pod \"3c196508-b0b5-44eb-8be9-190309a6184e\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") "
Oct 30 13:57:13.325244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.325123    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-maas-api-tls\") pod \"3c196508-b0b5-44eb-8be9-190309a6184e\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") "
Oct 30 13:57:13.325244 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.325199    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-metrics-certs\") pod \"3c196508-b0b5-44eb-8be9-190309a6184e\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") "
Oct 30 13:57:13.325410 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.325244    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c196508-b0b5-44eb-8be9-190309a6184e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c196508-b0b5-44eb-8be9-190309a6184e-openshift-service-ca\") pod \"3c196508-b0b5-44eb-8be9-190309a6184e\" (UID: \"3c196508-b0b5-44eb-8be9-190309a6184e\") "
Oct 30 13:57:13.325627 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.325600    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c196508-b0b5-44eb-8be9-190309a6184e-openshift-service-ca" pod "3c196508-b0b5-44eb-8be9-190309a6184e" (UID: "3c196508-b0b5-44eb-8be9-190309a6184e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:13.327511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.327489    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-maas-api-tls" pod "3c196508-b0b5-44eb-8be9-190309a6184e" (UID: "3c196508-b0b5-44eb-8be9-190309a6184e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:13.327600 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.327508    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c196508-b0b5-44eb-8be9-190309a6184e-kube-api-access-djdst" pod "3c196508-b0b5-44eb-8be9-190309a6184e" (UID: "3c196508-b0b5-44eb-8be9-190309a6184e"). InnerVolumeSpecName "kube-api-access-djdst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:13.327931 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.327914    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-metrics-certs" pod "3c196508-b0b5-44eb-8be9-190309a6184e" (UID: "3c196508-b0b5-44eb-8be9-190309a6184e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:13.426774 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.426688    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djdst\" (UniqueName: \"kubernetes.io/projected/3c196508-b0b5-44eb-8be9-190309a6184e-kube-api-access-djdst\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:13.426774 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.426733    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:13.426774 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.426755    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c196508-b0b5-44eb-8be9-190309a6184e-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:13.427189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:13.426774    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c196508-b0b5-44eb-8be9-190309a6184e-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:14.073087 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:14.073051    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7562f85-16f9-466c-8814-42b7c8307db1" path="/var/lib/kubelet/pods/b7562f85-16f9-466c-8814-42b7c8307db1/volumes"
Oct 30 13:57:14.073511 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:14.073497    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc86c779-342a-4ad1-bbe5-8a3cda3983a3" path="/var/lib/kubelet/pods/bc86c779-342a-4ad1-bbe5-8a3cda3983a3/volumes"
Oct 30 13:57:14.180979 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:14.180945    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m" event={"ID":"3c196508-b0b5-44eb-8be9-190309a6184e","Type":"ContainerDied","Data":"caadb2f7c45d3def5e5eab13a477cef9024d79f28250e1208e2a2f86dca54bd2"}
Oct 30 13:57:14.181131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:14.180997    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"
Oct 30 13:57:14.181131 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:14.181024    4429 scope.go:122] "RemoveContainer" containerID="dab3f4768e231ef324d3e4e8adbf5870d94ddb17706f0a80421cd868ccc5cf12"
Oct 30 13:57:14.241150 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:14.241124    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"]
Oct 30 13:57:14.254503 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:14.254481    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-451a21-797f578f9-znw4m"]
Oct 30 13:57:15.199549 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.199506    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"]
Oct 30 13:57:15.200014 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.199763    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs" podUID="22b81bf5-5279-44b4-a512-a1641b4f6a62" containerName="maas-api" containerID="cri-o://598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b" gracePeriod=30
Oct 30 13:57:15.452139 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.452083    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 13:57:15.550975 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.550945    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b81bf5-5279-44b4-a512-a1641b4f6a62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b81bf5-5279-44b4-a512-a1641b4f6a62-openshift-service-ca\") pod \"22b81bf5-5279-44b4-a512-a1641b4f6a62\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") "
Oct 30 13:57:15.551177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.551007    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b81bf5-5279-44b4-a512-a1641b4f6a62-kube-api-access-l4mcl\" (UniqueName: \"kubernetes.io/projected/22b81bf5-5279-44b4-a512-a1641b4f6a62-kube-api-access-l4mcl\") pod \"22b81bf5-5279-44b4-a512-a1641b4f6a62\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") "
Oct 30 13:57:15.551177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.551051    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-metrics-certs\") pod \"22b81bf5-5279-44b4-a512-a1641b4f6a62\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") "
Oct 30 13:57:15.551177 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.551089    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-maas-api-tls\") pod \"22b81bf5-5279-44b4-a512-a1641b4f6a62\" (UID: \"22b81bf5-5279-44b4-a512-a1641b4f6a62\") "
Oct 30 13:57:15.551423 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.551263    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b81bf5-5279-44b4-a512-a1641b4f6a62-openshift-service-ca" pod "22b81bf5-5279-44b4-a512-a1641b4f6a62" (UID: "22b81bf5-5279-44b4-a512-a1641b4f6a62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:15.553400 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.553373    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-metrics-certs" pod "22b81bf5-5279-44b4-a512-a1641b4f6a62" (UID: "22b81bf5-5279-44b4-a512-a1641b4f6a62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:15.553512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.553398    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-maas-api-tls" pod "22b81bf5-5279-44b4-a512-a1641b4f6a62" (UID: "22b81bf5-5279-44b4-a512-a1641b4f6a62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:15.553512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.553440    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b81bf5-5279-44b4-a512-a1641b4f6a62-kube-api-access-l4mcl" pod "22b81bf5-5279-44b4-a512-a1641b4f6a62" (UID: "22b81bf5-5279-44b4-a512-a1641b4f6a62"). InnerVolumeSpecName "kube-api-access-l4mcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:15.652189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.652144    4429 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-metrics-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:15.652189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.652175    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b81bf5-5279-44b4-a512-a1641b4f6a62-maas-api-tls\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:15.652189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.652185    4429 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b81bf5-5279-44b4-a512-a1641b4f6a62-openshift-service-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:15.652189 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:15.652195    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4mcl\" (UniqueName: \"kubernetes.io/projected/22b81bf5-5279-44b4-a512-a1641b4f6a62-kube-api-access-l4mcl\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:16.073643 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.073608    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c196508-b0b5-44eb-8be9-190309a6184e" path="/var/lib/kubelet/pods/3c196508-b0b5-44eb-8be9-190309a6184e/volumes"
Oct 30 13:57:16.194027 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.193993    4429 generic.go:356] "Generic (PLEG): container finished" podID="22b81bf5-5279-44b4-a512-a1641b4f6a62" containerID="598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b" exitCode=0
Oct 30 13:57:16.194205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.194052    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"
Oct 30 13:57:16.194205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.194070    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs" event={"ID":"22b81bf5-5279-44b4-a512-a1641b4f6a62","Type":"ContainerDied","Data":"598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b"}
Oct 30 13:57:16.194205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.194123    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs" event={"ID":"22b81bf5-5279-44b4-a512-a1641b4f6a62","Type":"ContainerDied","Data":"5cda3c055452c98012d5306398db467c67ba365b33a504b5dadcb52b4fe2be0a"}
Oct 30 13:57:16.194205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.194148    4429 scope.go:122] "RemoveContainer" containerID="598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b"
Oct 30 13:57:16.203793 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.203563    4429 scope.go:122] "RemoveContainer" containerID="598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b"
Oct 30 13:57:16.204128 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:16.203916    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b\": container with ID starting with 598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b not found: ID does not exist" containerID="598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b"
Oct 30 13:57:16.204128 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.203950    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b"} err="failed to get container status \"598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b\": rpc error: code = NotFound desc = could not find container \"598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b\": container with ID starting with 598af5e954f6d85c5dddc570cc93a9afc2121717ab71169133079c3dc6c9a74b not found: ID does not exist"
Oct 30 13:57:16.256896 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.256869    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"]
Oct 30 13:57:16.262914 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:16.262891    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7a8b03-cdcfcf96c-cw5bs"]
Oct 30 13:57:18.073178 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:18.073145    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b81bf5-5279-44b4-a512-a1641b4f6a62" path="/var/lib/kubelet/pods/22b81bf5-5279-44b4-a512-a1641b4f6a62/volumes"
Oct 30 13:57:22.102711 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:22.102679    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"]
Oct 30 13:57:22.103226 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:22.102880    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy" containerID="cri-o://c82d3a744ab0596d8eb2cb4abf8e07060eeaa24107c7895bd48c838647dd09c7" gracePeriod=30
Oct 30 13:57:27.261642 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.261608    4429 generic.go:356] "Generic (PLEG): container finished" podID="d920225c-2d94-4464-98af-fa12dc2428fd" containerID="c82d3a744ab0596d8eb2cb4abf8e07060eeaa24107c7895bd48c838647dd09c7" exitCode=0
Oct 30 13:57:27.262093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.261736    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" event={"ID":"d920225c-2d94-4464-98af-fa12dc2428fd","Type":"ContainerDied","Data":"c82d3a744ab0596d8eb2cb4abf8e07060eeaa24107c7895bd48c838647dd09c7"}
Oct 30 13:57:27.262093 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.261835    4429 scope.go:122] "RemoveContainer" containerID="d85949087494873ab4cd8bacb282aedb466668f17aa5e1368437d8f926f9e800"
Oct 30 13:57:27.363329 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.363302    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:57:27.463126 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463099    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-certs\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463145    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-kube-api-access-9mphp\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-kube-api-access-9mphp\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463166    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-istio-token\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-istio-token\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463200    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d920225c-2d94-4464-98af-fa12dc2428fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d920225c-2d94-4464-98af-fa12dc2428fd-istiod-ca-cert\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463317 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463245    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-envoy\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463421    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-certs" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:27.463542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463449    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-credential-socket\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463504    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-socket\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463582    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d920225c-2d94-4464-98af-fa12dc2428fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d920225c-2d94-4464-98af-fa12dc2428fd-istio-podinfo\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463616    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d920225c-2d94-4464-98af-fa12dc2428fd-istiod-ca-cert" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:27.463713 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463625    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-data\") pod \"d920225c-2d94-4464-98af-fa12dc2428fd\" (UID: \"d920225c-2d94-4464-98af-fa12dc2428fd\") "
Oct 30 13:57:27.463865 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463735    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-credential-socket" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:27.463865 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463757    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-socket" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:27.464008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.463990    4429 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-credential-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.464076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.464014    4429 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-socket\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.464076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.464031    4429 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-workload-certs\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.464076 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.464046    4429 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d920225c-2d94-4464-98af-fa12dc2428fd-istiod-ca-cert\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.464215 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.464076    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-data" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:27.465481 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.465461    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-kube-api-access-9mphp" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "kube-api-access-9mphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:27.465925 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.465905    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d920225c-2d94-4464-98af-fa12dc2428fd-istio-podinfo" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:57:27.466082 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.466061    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-envoy" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:27.466148 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.466066    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-istio-token" pod "d920225c-2d94-4464-98af-fa12dc2428fd" (UID: "d920225c-2d94-4464-98af-fa12dc2428fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:27.565533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.565501    4429 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-data\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.565533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.565528    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mphp\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-kube-api-access-9mphp\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.565533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.565541    4429 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d920225c-2d94-4464-98af-fa12dc2428fd-istio-token\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.565795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.565549    4429 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d920225c-2d94-4464-98af-fa12dc2428fd-istio-envoy\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:27.565795 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:27.565557    4429 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d920225c-2d94-4464-98af-fa12dc2428fd-istio-podinfo\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:28.267841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:28.267808    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4" event={"ID":"d920225c-2d94-4464-98af-fa12dc2428fd","Type":"ContainerDied","Data":"f0b24e130136e6ed881ac64b17fefe6a8c4c011ce2c764f707c89e5e34705126"}
Oct 30 13:57:28.267841 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:28.267811    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"
Oct 30 13:57:28.268394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:28.267872    4429 scope.go:122] "RemoveContainer" containerID="c82d3a744ab0596d8eb2cb4abf8e07060eeaa24107c7895bd48c838647dd09c7"
Oct 30 13:57:28.337963 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:28.337883    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"]
Oct 30 13:57:28.351529 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:28.351501    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-451a21-openshift-default-c48d9fc87-spvw4"]
Oct 30 13:57:30.073264 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:30.073230    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" path="/var/lib/kubelet/pods/d920225c-2d94-4464-98af-fa12dc2428fd/volumes"
Oct 30 13:57:33.372148 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:33.372113    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"]
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.300510    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"]
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301090    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b724c857-22c4-41c1-a1f4-85548435e237" containerName="payload-processing"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301110    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b724c857-22c4-41c1-a1f4-85548435e237" containerName="payload-processing"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301126    4429 cpu_manager.go:414] "containerMap: removing container" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301134    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301144    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b0d701f4-05ec-48c8-898b-d2b954e8f368" containerName="payload-pre-processing"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301152    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d701f4-05ec-48c8-898b-d2b954e8f368" containerName="payload-pre-processing"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301165    4429 cpu_manager.go:414] "containerMap: removing container" podUID="bc7d3080-95f3-4fca-9680-24a2b982f3d0" containerName="payload-pre-processing"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301173    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7d3080-95f3-4fca-9680-24a2b982f3d0" containerName="payload-pre-processing"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301190    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b7562f85-16f9-466c-8814-42b7c8307db1" containerName="payload-processing"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301197    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7562f85-16f9-466c-8814-42b7c8307db1" containerName="payload-processing"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301207    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main"
Oct 30 13:57:36.300542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301214    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301226    4429 cpu_manager.go:414] "containerMap: removing container" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301233    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301242    4429 cpu_manager.go:414] "containerMap: removing container" podUID="171bd558-2e74-4d2c-ad34-41118c9c26ef" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301248    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171bd558-2e74-4d2c-ad34-41118c9c26ef" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301260    4429 cpu_manager.go:414] "containerMap: removing container" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301268    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301277    4429 cpu_manager.go:414] "containerMap: removing container" podUID="c74aba3a-269e-4d9c-88a3-4b224d852c36" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301305    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74aba3a-269e-4d9c-88a3-4b224d852c36" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301316    4429 cpu_manager.go:414] "containerMap: removing container" podUID="e361b56b-e61d-4c0f-a915-da0415767b13" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301324    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e361b56b-e61d-4c0f-a915-da0415767b13" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301337    4429 cpu_manager.go:414] "containerMap: removing container" podUID="6be5a43a-2320-4e63-821a-e15e67da2f81" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301344    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be5a43a-2320-4e63-821a-e15e67da2f81" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301357    4429 cpu_manager.go:414] "containerMap: removing container" podUID="eb361f7e-e214-47b7-bc86-e37708877f24" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301364    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb361f7e-e214-47b7-bc86-e37708877f24" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301377    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9dfa2574-66ba-4023-b91f-37ed39e299e1" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301384    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dfa2574-66ba-4023-b91f-37ed39e299e1" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301399    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b1b1df1d-e2a2-4fe2-8598-49573af73613" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301406    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b1df1d-e2a2-4fe2-8598-49573af73613" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301414    4429 cpu_manager.go:414] "containerMap: removing container" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301421    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301434    4429 cpu_manager.go:414] "containerMap: removing container" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301440    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301455    4429 cpu_manager.go:414] "containerMap: removing container" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301461    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301471    4429 cpu_manager.go:414] "containerMap: removing container" podUID="bc24016f-ddfd-4c79-a015-4684b56b817f" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301478    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc24016f-ddfd-4c79-a015-4684b56b817f" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301493    4429 cpu_manager.go:414] "containerMap: removing container" podUID="22b81bf5-5279-44b4-a512-a1641b4f6a62" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301500    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b81bf5-5279-44b4-a512-a1641b4f6a62" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301511    4429 cpu_manager.go:414] "containerMap: removing container" podUID="bc289b98-9450-489e-975a-b2d6ba7f5b9f" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301518    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc289b98-9450-489e-975a-b2d6ba7f5b9f" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301528    4429 cpu_manager.go:414] "containerMap: removing container" podUID="cabcd6da-a086-4864-b2e7-0d41500b6c83" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301534    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabcd6da-a086-4864-b2e7-0d41500b6c83" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301548    4429 cpu_manager.go:414] "containerMap: removing container" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301556    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301565    4429 cpu_manager.go:414] "containerMap: removing container" podUID="62d4287c-8aba-4241-9d46-66577c5dc2b6" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301572    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d4287c-8aba-4241-9d46-66577c5dc2b6" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301583    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301590    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301602    4429 cpu_manager.go:414] "containerMap: removing container" podUID="33e2e18d-e039-43aa-8822-52590f759cfd" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301609    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e2e18d-e039-43aa-8822-52590f759cfd" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301620    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b3da904d-79b0-4f10-b122-fb0720de4008" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301627    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3da904d-79b0-4f10-b122-fb0720de4008" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301639    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301646    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301655    4429 cpu_manager.go:414] "containerMap: removing container" podUID="4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301662    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301682    4429 cpu_manager.go:414] "containerMap: removing container" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301690    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301703    4429 cpu_manager.go:414] "containerMap: removing container" podUID="03680b2d-91b9-40e8-a3c5-5061188362b4" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301711    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03680b2d-91b9-40e8-a3c5-5061188362b4" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301721    4429 cpu_manager.go:414] "containerMap: removing container" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301728    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301739    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301746    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301756    4429 cpu_manager.go:414] "containerMap: removing container" podUID="60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301762    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301772    4429 cpu_manager.go:414] "containerMap: removing container" podUID="3c196508-b0b5-44eb-8be9-190309a6184e" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301779    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c196508-b0b5-44eb-8be9-190309a6184e" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301791    4429 cpu_manager.go:414] "containerMap: removing container" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301798    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301806    4429 cpu_manager.go:414] "containerMap: removing container" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301813    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301822    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301829    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301839    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301848    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301861    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9017cc61-e4f0-403a-90f9-a0bebe754590" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301867    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9017cc61-e4f0-403a-90f9-a0bebe754590" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301877    4429 cpu_manager.go:414] "containerMap: removing container" podUID="9830a470-d928-4baf-b5b0-da84535d7cd8" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301883    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9830a470-d928-4baf-b5b0-da84535d7cd8" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301895    4429 cpu_manager.go:414] "containerMap: removing container" podUID="591c870e-000d-4d4d-a1a6-1f52be1a23da" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301901    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591c870e-000d-4d4d-a1a6-1f52be1a23da" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301910    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301916    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301926    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301933    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301942    4429 cpu_manager.go:414] "containerMap: removing container" podUID="d75912f6-090d-4853-8c85-c714af76d1f4" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301950    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75912f6-090d-4853-8c85-c714af76d1f4" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301959    4429 cpu_manager.go:414] "containerMap: removing container" podUID="97305157-a41d-465e-8e94-1aa9dc38a166" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301968    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97305157-a41d-465e-8e94-1aa9dc38a166" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301977    4429 cpu_manager.go:414] "containerMap: removing container" podUID="1ca25a94-48bd-4048-9955-09b137fbefe4" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301982    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca25a94-48bd-4048-9955-09b137fbefe4" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301989    4429 cpu_manager.go:414] "containerMap: removing container" podUID="94e10d3e-d608-45bb-bfc5-f8905b852890" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301993    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e10d3e-d608-45bb-bfc5-f8905b852890" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.301999    4429 cpu_manager.go:414] "containerMap: removing container" podUID="88228612-d786-4f85-9cea-076c8db124a4" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302003    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88228612-d786-4f85-9cea-076c8db124a4" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302012    4429 cpu_manager.go:414] "containerMap: removing container" podUID="3c5b7fde-8638-4360-b709-58c7f03ca5e2" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302019    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5b7fde-8638-4360-b709-58c7f03ca5e2" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302026    4429 cpu_manager.go:414] "containerMap: removing container" podUID="edfc88d7-a686-4947-af56-ff979b8d603d" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302032    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edfc88d7-a686-4947-af56-ff979b8d603d" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302041    4429 cpu_manager.go:414] "containerMap: removing container" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302048    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302059    4429 cpu_manager.go:414] "containerMap: removing container" podUID="eba1899f-c0d5-4484-aa5b-eb878b13b571" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302065    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba1899f-c0d5-4484-aa5b-eb878b13b571" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302074    4429 cpu_manager.go:414] "containerMap: removing container" podUID="bc86c779-342a-4ad1-bbe5-8a3cda3983a3" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302080    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc86c779-342a-4ad1-bbe5-8a3cda3983a3" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302181    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5b7fde-8638-4360-b709-58c7f03ca5e2" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302194    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="edfc88d7-a686-4947-af56-ff979b8d603d" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302206    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302216    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba1899f-c0d5-4484-aa5b-eb878b13b571" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302227    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc86c779-342a-4ad1-bbe5-8a3cda3983a3" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302237    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b724c857-22c4-41c1-a1f4-85548435e237" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302246    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302254    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d701f4-05ec-48c8-898b-d2b954e8f368" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302266    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7d3080-95f3-4fca-9680-24a2b982f3d0" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302276    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302306    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7562f85-16f9-466c-8814-42b7c8307db1" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302315    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4a09b0-d45a-4c6b-9c22-80c35693bc8e" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302322    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebcbb19a-e0e3-46d6-adcf-405ca61eb41b" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302330    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="171bd558-2e74-4d2c-ad34-41118c9c26ef" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302340    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302347    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74aba3a-269e-4d9c-88a3-4b224d852c36" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302356    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="e361b56b-e61d-4c0f-a915-da0415767b13" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302365    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be5a43a-2320-4e63-821a-e15e67da2f81" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302375    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb361f7e-e214-47b7-bc86-e37708877f24" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302385    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dfa2574-66ba-4023-b91f-37ed39e299e1" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302393    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b1df1d-e2a2-4fe2-8598-49573af73613" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302401    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e886c2e-e044-4f32-af32-9945c531b70f" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302410    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0507ec-f598-4e58-ad65-2a0afb295eaf" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302419    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc24016f-ddfd-4c79-a015-4684b56b817f" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302428    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b81bf5-5279-44b4-a512-a1641b4f6a62" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302438    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc289b98-9450-489e-975a-b2d6ba7f5b9f" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302447    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabcd6da-a086-4864-b2e7-0d41500b6c83" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302457    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302465    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d4287c-8aba-4241-9d46-66577c5dc2b6" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302474    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1494f395-8b0b-472c-bf57-645835078d44" containerName="main"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302482    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e2e18d-e039-43aa-8822-52590f759cfd" containerName="payload-pre-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302491    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3da904d-79b0-4f10-b122-fb0720de4008" containerName="payload-processing"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302502    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302511    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1282ee-48bd-49fa-8af5-5c22dc7abfc6" containerName="maas-api"
Oct 30 13:57:36.302421 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302520    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64ff59a-c76d-41ab-9c2a-55c717e626cf" containerName="istio-proxy"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302529    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="03680b2d-91b9-40e8-a3c5-5061188362b4" containerName="payload-pre-processing"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302540    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd7c4dc-0253-42f2-a62d-3af6b6de9cf1" containerName="main"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302549    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5b7987-fc95-4e40-9acf-6f4b52b68a8d" containerName="payload-pre-processing"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302557    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e4eccb-a6cc-462f-8a8c-e6d5db3f00c9" containerName="maas-api"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302565    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c196508-b0b5-44eb-8be9-190309a6184e" containerName="maas-api"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302571    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="f271de4a-9331-4b94-8d60-24a3eeb2df0c" containerName="main"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302577    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="b327a616-e506-492f-88b7-66e4143df16a" containerName="istio-proxy"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302583    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302588    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d920225c-2d94-4464-98af-fa12dc2428fd" containerName="istio-proxy"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302593    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9017cc61-e4f0-403a-90f9-a0bebe754590" containerName="maas-api"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302597    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="9830a470-d928-4baf-b5b0-da84535d7cd8" containerName="main"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302603    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="591c870e-000d-4d4d-a1a6-1f52be1a23da" containerName="payload-pre-processing"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302611    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302619    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f1e8a2-eb2a-4e11-8923-22a34e177c4a" containerName="payload-processing"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302629    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75912f6-090d-4853-8c85-c714af76d1f4" containerName="payload-processing"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302636    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="97305157-a41d-465e-8e94-1aa9dc38a166" containerName="payload-processing"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302644    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca25a94-48bd-4048-9955-09b137fbefe4" containerName="payload-processing"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302648    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e10d3e-d608-45bb-bfc5-f8905b852890" containerName="payload-pre-processing"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.302654    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="88228612-d786-4f85-9cea-076c8db124a4" containerName="maas-api"
Oct 30 13:57:36.309429 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.305858    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.324546 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.324509    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"]
Oct 30 13:57:36.344435 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.344403    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4js\" (UniqueName: \"kubernetes.io/projected/05901a59-64d2-4ebb-aa7f-fd219214fa39-kube-api-access-7t4js\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.344566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.344477    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.344566 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.344506    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.401832 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.401803    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:57:36.401990 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.401943    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f8hh2\""
Oct 30 13:57:36.445635 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.445608    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4js\" (UniqueName: \"kubernetes.io/projected/05901a59-64d2-4ebb-aa7f-fd219214fa39-kube-api-access-7t4js\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.445776 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.445673    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.445776 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.445702    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.446319 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.446276    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.448313 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.448277    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.470633 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.470610    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4js\" (UniqueName: \"kubernetes.io/projected/05901a59-64d2-4ebb-aa7f-fd219214fa39-kube-api-access-7t4js\") pod \"maas-api-revoke-sub-5f68427a-zrqqt\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.619201 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.619133    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:36.920913 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:36.920880    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"]
Oct 30 13:57:36.922018 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:57:36.921995    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05901a59_64d2_4ebb_aa7f_fd219214fa39.slice/crio-6d8683399bbf3b4c15c17887c1e75d4147f3d59d39542e66f5343d67118ed35e WatchSource:0}: Error finding container 6d8683399bbf3b4c15c17887c1e75d4147f3d59d39542e66f5343d67118ed35e: Status 404 returned error can't find the container with id 6d8683399bbf3b4c15c17887c1e75d4147f3d59d39542e66f5343d67118ed35e
Oct 30 13:57:37.311150 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:37.311117    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt" event={"ID":"05901a59-64d2-4ebb-aa7f-fd219214fa39","Type":"ContainerStarted","Data":"1b6981a30231a5a38253e5e9d4d116bfe5186e252b0bd999d22315abc5291862"}
Oct 30 13:57:37.311533 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:37.311169    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt" event={"ID":"05901a59-64d2-4ebb-aa7f-fd219214fa39","Type":"ContainerStarted","Data":"6d8683399bbf3b4c15c17887c1e75d4147f3d59d39542e66f5343d67118ed35e"}
Oct 30 13:57:37.355873 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:37.355823    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt" podStartSLOduration=1.355809458 podStartE2EDuration="1.355809458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:37.355154482 +0000 UTC m=+5115.410648305" watchObservedRunningTime="2026-10-30 13:57:37.355809458 +0000 UTC m=+5115.411303281"
Oct 30 13:57:38.317341 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.317306    4429 generic.go:356] "Generic (PLEG): container finished" podID="05901a59-64d2-4ebb-aa7f-fd219214fa39" containerID="1b6981a30231a5a38253e5e9d4d116bfe5186e252b0bd999d22315abc5291862" exitCode=0
Oct 30 13:57:38.317737 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.317379    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt" event={"ID":"05901a59-64d2-4ebb-aa7f-fd219214fa39","Type":"ContainerDied","Data":"1b6981a30231a5a38253e5e9d4d116bfe5186e252b0bd999d22315abc5291862"}
Oct 30 13:57:38.373297 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.373242    4429 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2" podUID="bf3a2953-1200-438e-a224-6259a412c53f" containerName="payload-pre-processing" containerID="cri-o://7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35" gracePeriod=25
Oct 30 13:57:38.614148 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.614117    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 13:57:38.662771 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.662736    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3a2953-1200-438e-a224-6259a412c53f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3a2953-1200-438e-a224-6259a412c53f-plugins-config-volume\") pod \"bf3a2953-1200-438e-a224-6259a412c53f\" (UID: \"bf3a2953-1200-438e-a224-6259a412c53f\") "
Oct 30 13:57:38.662949 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.662895    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3a2953-1200-438e-a224-6259a412c53f-kube-api-access-frf8j\" (UniqueName: \"kubernetes.io/projected/bf3a2953-1200-438e-a224-6259a412c53f-kube-api-access-frf8j\") pod \"bf3a2953-1200-438e-a224-6259a412c53f\" (UID: \"bf3a2953-1200-438e-a224-6259a412c53f\") "
Oct 30 13:57:38.663234 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.663206    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3a2953-1200-438e-a224-6259a412c53f-plugins-config-volume" pod "bf3a2953-1200-438e-a224-6259a412c53f" (UID: "bf3a2953-1200-438e-a224-6259a412c53f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:38.665492 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.665457    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3a2953-1200-438e-a224-6259a412c53f-kube-api-access-frf8j" pod "bf3a2953-1200-438e-a224-6259a412c53f" (UID: "bf3a2953-1200-438e-a224-6259a412c53f"). InnerVolumeSpecName "kube-api-access-frf8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:38.764544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.764508    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frf8j\" (UniqueName: \"kubernetes.io/projected/bf3a2953-1200-438e-a224-6259a412c53f-kube-api-access-frf8j\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:38.764544 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:38.764535    4429 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3a2953-1200-438e-a224-6259a412c53f-plugins-config-volume\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:39.324072 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.324038    4429 generic.go:356] "Generic (PLEG): container finished" podID="bf3a2953-1200-438e-a224-6259a412c53f" containerID="7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35" exitCode=0
Oct 30 13:57:39.324535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.324102    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"
Oct 30 13:57:39.324535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.324125    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2" event={"ID":"bf3a2953-1200-438e-a224-6259a412c53f","Type":"ContainerDied","Data":"7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35"}
Oct 30 13:57:39.324535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.324199    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2" event={"ID":"bf3a2953-1200-438e-a224-6259a412c53f","Type":"ContainerDied","Data":"f5061124e200e0708d342e83c2fac37fe9f01c7261b6f7b8b0dd0bd1de608c59"}
Oct 30 13:57:39.324535 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.324221    4429 scope.go:122] "RemoveContainer" containerID="7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35"
Oct 30 13:57:39.333722 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.333705    4429 scope.go:122] "RemoveContainer" containerID="7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35"
Oct 30 13:57:39.333999 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:39.333979    4429 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35\": container with ID starting with 7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35 not found: ID does not exist" containerID="7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35"
Oct 30 13:57:39.334068 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.334009    4429 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35"} err="failed to get container status \"7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35\": rpc error: code = NotFound desc = could not find container \"7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35\": container with ID starting with 7bf0c008321e00fd3be34aa2aff5e06fc70d41ae311f852f548d267b3cc3ab35 not found: ID does not exist"
Oct 30 13:57:39.384397 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.384366    4429 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"]
Oct 30 13:57:39.398957 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.398925    4429 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-551dd1-7c4f8b8f55-swrq2"]
Oct 30 13:57:39.451458 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.451434    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:39.471888 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.471867    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-openshift-ca\") pod \"05901a59-64d2-4ebb-aa7f-fd219214fa39\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") "
Oct 30 13:57:39.472011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.471957    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05901a59-64d2-4ebb-aa7f-fd219214fa39-kube-api-access-7t4js\" (UniqueName: \"kubernetes.io/projected/05901a59-64d2-4ebb-aa7f-fd219214fa39-kube-api-access-7t4js\") pod \"05901a59-64d2-4ebb-aa7f-fd219214fa39\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") "
Oct 30 13:57:39.472011 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.471985    4429 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-xks-ca\") pod \"05901a59-64d2-4ebb-aa7f-fd219214fa39\" (UID: \"05901a59-64d2-4ebb-aa7f-fd219214fa39\") "
Oct 30 13:57:39.472200 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.472179    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-openshift-ca" pod "05901a59-64d2-4ebb-aa7f-fd219214fa39" (UID: "05901a59-64d2-4ebb-aa7f-fd219214fa39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:39.474103 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.474078    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05901a59-64d2-4ebb-aa7f-fd219214fa39-kube-api-access-7t4js" pod "05901a59-64d2-4ebb-aa7f-fd219214fa39" (UID: "05901a59-64d2-4ebb-aa7f-fd219214fa39"). InnerVolumeSpecName "kube-api-access-7t4js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:39.474207 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.474108    4429 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-xks-ca" pod "05901a59-64d2-4ebb-aa7f-fd219214fa39" (UID: "05901a59-64d2-4ebb-aa7f-fd219214fa39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:39.573245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.573212    4429 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t4js\" (UniqueName: \"kubernetes.io/projected/05901a59-64d2-4ebb-aa7f-fd219214fa39-kube-api-access-7t4js\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:39.573245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.573242    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-xks-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:39.573245 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:39.573251    4429 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05901a59-64d2-4ebb-aa7f-fd219214fa39-maas-api-openshift-ca\") on node \"ip-10-0-136-246.ec2.internal\" DevicePath \"\""
Oct 30 13:57:40.073868 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:40.073833    4429 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3a2953-1200-438e-a224-6259a412c53f" path="/var/lib/kubelet/pods/bf3a2953-1200-438e-a224-6259a412c53f/volumes"
Oct 30 13:57:40.331853 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:40.331772    4429 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt"
Oct 30 13:57:40.332359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:40.331774    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f68427a-zrqqt" event={"ID":"05901a59-64d2-4ebb-aa7f-fd219214fa39","Type":"ContainerDied","Data":"6d8683399bbf3b4c15c17887c1e75d4147f3d59d39542e66f5343d67118ed35e"}
Oct 30 13:57:40.332359 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:57:40.332339    4429 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8683399bbf3b4c15c17887c1e75d4147f3d59d39542e66f5343d67118ed35e"
Oct 30 13:57:42.370934 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:57:42.370900    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:58:03.733824 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:03.733788    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ddfbb87d4-bkzx2_45e3a6bf-a04a-4001-8036-28397efa666c/manager/0.log"
Oct 30 13:58:04.396999 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:04.396970    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rcx5n_c383851d-6fc2-4523-9a52-b603e4149172/proxy/0.log"
Oct 30 13:58:12.371531 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:58:12.371496    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:58:17.032848 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:17.032779    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ddfbb87d4-bkzx2_45e3a6bf-a04a-4001-8036-28397efa666c/manager/0.log"
Oct 30 13:58:17.658499 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:17.658472    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rcx5n_c383851d-6fc2-4523-9a52-b603e4149172/proxy/0.log"
Oct 30 13:58:18.428880 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:18.428844    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78747dffff-qqr9g_f1f55165-93cb-4197-935e-a6f2e6cbc44a/maas-api/0.log"
Oct 30 13:58:20.185078 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:20.185051    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5f68427a-zrqqt_05901a59-64d2-4ebb-aa7f-fd219214fa39/revoke-keys/0.log"
Oct 30 13:58:26.145280 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:26.145248    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59688578c6-ds4tj_2d39d81d-4605-4dec-8b57-bb683a61a741/kube-auth-proxy/0.log"
Oct 30 13:58:27.147267 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:27.147239    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qvm42_c513e96c-4db5-43e5-bbd1-eda409197969/istio-proxy/0.log"
Oct 30 13:58:29.550970 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:29.550933    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9d754_037e455b-6b2a-4cf5-a8e8-0d606071d12c/main/0.log"
Oct 30 13:58:29.751371 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:29.751332    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-gjcgq_cd0b29e6-33d5-4327-836f-57c1eea4e635/main/0.log"
Oct 30 13:58:30.136583 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:30.136550    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-c6qkj_1faa86f9-abf0-4dd5-ab7e-c087ffe1ea7e/main/0.log"
Oct 30 13:58:42.371055 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:58:42.371017    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:58:46.352394 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:46.352364    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6h57s_a3349844-7e31-4934-8c31-2a208adbf1a5/global-pull-secret-syncer/0.log"
Oct 30 13:58:48.025539 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:48.025511    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xz8gl_cee19c40-c94c-45cd-8deb-16e8a965e053/konnectivity-agent/0.log"
Oct 30 13:58:48.517396 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:48.517365    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-246.ec2.internal_e9b5f11662f569a9096e79917d2f933f/haproxy/0.log"
Oct 30 13:58:57.905108 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905075    4429 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"]
Oct 30 13:58:57.905556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905470    4429 cpu_manager.go:414] "containerMap: removing container" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy"
Oct 30 13:58:57.905556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905482    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6206414c-fa76-42c9-a13e-7d4a45fea76b" containerName="istio-proxy"
Oct 30 13:58:57.905556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905498    4429 cpu_manager.go:414] "containerMap: removing container" podUID="05901a59-64d2-4ebb-aa7f-fd219214fa39" containerName="revoke-keys"
Oct 30 13:58:57.905556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905503    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05901a59-64d2-4ebb-aa7f-fd219214fa39" containerName="revoke-keys"
Oct 30 13:58:57.905556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905516    4429 cpu_manager.go:414] "containerMap: removing container" podUID="bf3a2953-1200-438e-a224-6259a412c53f" containerName="payload-pre-processing"
Oct 30 13:58:57.905556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905524    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3a2953-1200-438e-a224-6259a412c53f" containerName="payload-pre-processing"
Oct 30 13:58:57.905556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905544    4429 cpu_manager.go:414] "containerMap: removing container" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy"
Oct 30 13:58:57.905556 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905550    4429 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153040de-ee6d-4eb0-9333-5e7d0c36dc60" containerName="istio-proxy"
Oct 30 13:58:57.905839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905621    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="05901a59-64d2-4ebb-aa7f-fd219214fa39" containerName="revoke-keys"
Oct 30 13:58:57.905839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905630    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3a2953-1200-438e-a224-6259a412c53f" containerName="payload-pre-processing"
Oct 30 13:58:57.905839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905639    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6dd987-feca-427c-8f9b-724c899c2678" containerName="istio-proxy"
Oct 30 13:58:57.905839 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.905645    4429 memory_manager.go:365] "RemoveStaleState removing state" podUID="922d2e60-03b5-4ae7-b439-3b57d47d2902" containerName="istio-proxy"
Oct 30 13:58:57.908505 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.908487    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:57.927922 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.927894    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"]
Oct 30 13:58:57.940442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.940378    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z45hd\"/\"kube-root-ca.crt\""
Oct 30 13:58:57.940442 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.940369    4429 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z45hd\"/\"openshift-service-ca.crt\""
Oct 30 13:58:57.940630 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:57.940447    4429 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z45hd\"/\"default-dockercfg-ln669\""
Oct 30 13:58:58.015183 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.015151    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmdl\" (UniqueName: \"kubernetes.io/projected/289e1d3c-d09d-40db-8218-4aa9bb0a0868-kube-api-access-qmmdl\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.015384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.015202    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-lib-modules\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.015384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.015276    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-sys\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.015384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.015353    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-podres\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.015384 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.015373    4429 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-proc\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116445 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116417    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-lib-modules\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116454    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-sys\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116490    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-podres\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116513    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-proc\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116555    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-sys\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116617 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116559    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-lib-modules\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116630    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-proc\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116712    4429 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmdl\" (UniqueName: \"kubernetes.io/projected/289e1d3c-d09d-40db-8218-4aa9bb0a0868-kube-api-access-qmmdl\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.116796 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.116729    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289e1d3c-d09d-40db-8218-4aa9bb0a0868-podres\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.142008 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.141982    4429 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmdl\" (UniqueName: \"kubernetes.io/projected/289e1d3c-d09d-40db-8218-4aa9bb0a0868-kube-api-access-qmmdl\") pod \"perf-node-gather-daemonset-x6hnm\" (UID: \"289e1d3c-d09d-40db-8218-4aa9bb0a0868\") " pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.221840 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.221815    4429 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.518057 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.518032    4429 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"]
Oct 30 13:58:58.518679 ip-10-0-136-246 kubenswrapper[4429]: W1030 13:58:58.518654    4429 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod289e1d3c_d09d_40db_8218_4aa9bb0a0868.slice/crio-11940de1468928df9f7a5e789eed744fded9476494e79103a24b48f842125b3f WatchSource:0}: Error finding container 11940de1468928df9f7a5e789eed744fded9476494e79103a24b48f842125b3f: Status 404 returned error can't find the container with id 11940de1468928df9f7a5e789eed744fded9476494e79103a24b48f842125b3f
Oct 30 13:58:58.683940 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.683913    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm" event={"ID":"289e1d3c-d09d-40db-8218-4aa9bb0a0868","Type":"ContainerStarted","Data":"d1275a62ed3745fef533f833350ac40a638fda55b40be1e8ba63ab9bdfb93f61"}
Oct 30 13:58:58.684051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.683965    4429 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm" event={"ID":"289e1d3c-d09d-40db-8218-4aa9bb0a0868","Type":"ContainerStarted","Data":"11940de1468928df9f7a5e789eed744fded9476494e79103a24b48f842125b3f"}
Oct 30 13:58:58.684051 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.684011    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:58:58.747205 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.747158    4429 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm" podStartSLOduration=1.7471448280000001 podStartE2EDuration="1.747144828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:58.745033687 +0000 UTC m=+5196.800527514" watchObservedRunningTime="2026-10-30 13:58:58.747144828 +0000 UTC m=+5196.802638660"
Oct 30 13:58:58.866773 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:58.866699    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-69s5m_093ae6a9-190d-43f8-a883-8807b82d47c5/kube-state-metrics/0.log"
Oct 30 13:58:59.004521 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:59.004489    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-69s5m_093ae6a9-190d-43f8-a883-8807b82d47c5/kube-rbac-proxy-main/0.log"
Oct 30 13:58:59.128995 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:59.128924    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-69s5m_093ae6a9-190d-43f8-a883-8807b82d47c5/kube-rbac-proxy-self/0.log"
Oct 30 13:58:59.263147 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:59.263122    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57f9c5fc79-snqqs_067d351d-52ac-4f35-809f-99c8e9acf213/metrics-server/0.log"
Oct 30 13:58:59.937634 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:58:59.937523    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pq7wj_9c8a7124-855e-46a2-99fd-0eba3204c556/node-exporter/0.log"
Oct 30 13:59:00.063611 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:00.063580    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pq7wj_9c8a7124-855e-46a2-99fd-0eba3204c556/kube-rbac-proxy/0.log"
Oct 30 13:59:00.188696 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:00.188621    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pq7wj_9c8a7124-855e-46a2-99fd-0eba3204c556/init-textfile/0.log"
Oct 30 13:59:00.832316 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:00.832262    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6qvps_05a06931-11e7-4665-850e-4d6732c39faa/kube-rbac-proxy-main/0.log"
Oct 30 13:59:00.985380 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:00.985348    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6qvps_05a06931-11e7-4665-850e-4d6732c39faa/kube-rbac-proxy-self/0.log"
Oct 30 13:59:01.110342 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.110237    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ddfbb87d4-bkzx2_45e3a6bf-a04a-4001-8036-28397efa666c/manager/0.log"
Oct 30 13:59:01.123699 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.123672    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6qvps_05a06931-11e7-4665-850e-4d6732c39faa/openshift-state-metrics/0.log"
Oct 30 13:59:01.133075 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.133052    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ddfbb87d4-bkzx2_45e3a6bf-a04a-4001-8036-28397efa666c/manager/0.log"
Oct 30 13:59:01.261542 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.261514    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f058f46-6d9b-4469-b091-063d2f667260/prometheus/0.log"
Oct 30 13:59:01.396223 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.396143    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f058f46-6d9b-4469-b091-063d2f667260/config-reloader/0.log"
Oct 30 13:59:01.524709 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.524684    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f058f46-6d9b-4469-b091-063d2f667260/thanos-sidecar/0.log"
Oct 30 13:59:01.647444 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.647370    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f058f46-6d9b-4469-b091-063d2f667260/kube-rbac-proxy-web/0.log"
Oct 30 13:59:01.772512 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.772488    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f058f46-6d9b-4469-b091-063d2f667260/kube-rbac-proxy/0.log"
Oct 30 13:59:01.906059 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:01.905989    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f058f46-6d9b-4469-b091-063d2f667260/kube-rbac-proxy-thanos/0.log"
Oct 30 13:59:02.030495 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:02.030425    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f058f46-6d9b-4469-b091-063d2f667260/init-config-reloader/0.log"
Oct 30 13:59:02.427621 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:02.427592    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-8mtss_28d28355-8803-47b6-9cdd-106503274ebc/prometheus-operator-admission-webhook/0.log"
Oct 30 13:59:03.069249 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:03.069221    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rcx5n_c383851d-6fc2-4523-9a52-b603e4149172/proxy/0.log"
Oct 30 13:59:03.069783 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:03.069763    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rcx5n_c383851d-6fc2-4523-9a52-b603e4149172/proxy/0.log"
Oct 30 13:59:04.700036 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:04.700012    4429 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z45hd/perf-node-gather-daemonset-x6hnm"
Oct 30 13:59:06.235550 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:06.235515    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ddfbb87d4-bkzx2_45e3a6bf-a04a-4001-8036-28397efa666c/manager/0.log"
Oct 30 13:59:06.722565 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:06.722530    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ddfbb87d4-bkzx2_45e3a6bf-a04a-4001-8036-28397efa666c/manager/0.log"
Oct 30 13:59:08.072070 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:08.072042    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rcx5n_c383851d-6fc2-4523-9a52-b603e4149172/proxy/0.log"
Oct 30 13:59:08.757792 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:08.757763    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rcx5n_c383851d-6fc2-4523-9a52-b603e4149172/proxy/0.log"
Oct 30 13:59:11.369895 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:11.369860    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ddfbb87d4-bkzx2_45e3a6bf-a04a-4001-8036-28397efa666c/manager/0.log"
Oct 30 13:59:11.902146 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:11.902108    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7ddfbb87d4-bkzx2_45e3a6bf-a04a-4001-8036-28397efa666c/manager/0.log"
Oct 30 13:59:12.380924 ip-10-0-136-246 kubenswrapper[4429]: E1030 13:59:12.380877    4429 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gnl9r" containerName="extractor"
Oct 30 13:59:12.605092 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:12.605059    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fj9wk_7208585d-b1f1-4d3f-9348-2e4abaaa8e26/dns/0.log"
Oct 30 13:59:12.729590 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:12.729562    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fj9wk_7208585d-b1f1-4d3f-9348-2e4abaaa8e26/kube-rbac-proxy/0.log"
Oct 30 13:59:13.298787 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:13.298756    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rcx5n_c383851d-6fc2-4523-9a52-b603e4149172/proxy/0.log"
Oct 30 13:59:13.507603 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:13.507572    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59688578c6-ds4tj_2d39d81d-4605-4dec-8b57-bb683a61a741/kube-auth-proxy/0.log"
Oct 30 13:59:13.522454 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:13.522431    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fttst_4825a637-bc1a-4b66-acd1-c0081284ae23/dns-node-resolver/0.log"
Oct 30 13:59:13.836727 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:13.836698    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rcx5n_c383851d-6fc2-4523-9a52-b603e4149172/proxy/0.log"
Oct 30 13:59:13.962798 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:13.962773    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qvm42_c513e96c-4db5-43e5-bbd1-eda409197969/istio-proxy/0.log"
Oct 30 13:59:14.881006 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:14.880980    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67d678455d-64qcq_ae1fb865-d9e1-4a30-9241-4c0cc7c0ae83/registry/0.log"
Oct 30 13:59:14.989782 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:14.989749    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r8lf8_6bf513a7-048e-42d8-98fb-5b7adbc09011/node-ca/0.log"
Oct 30 13:59:15.623214 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:15.623185    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59688578c6-ds4tj_2d39d81d-4605-4dec-8b57-bb683a61a741/kube-auth-proxy/0.log"
Oct 30 13:59:16.024562 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:16.024531    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qvm42_c513e96c-4db5-43e5-bbd1-eda409197969/istio-proxy/0.log"
Oct 30 13:59:16.096368 ip-10-0-136-246 kubenswrapper[4429]: I1030 13:59:16.096345    4429 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59688578c6-ds4tj_2d39d81d-4605-4dec-8b57-bb683a61a741/kube-auth-proxy/0.log"